00000100:00080000:0.0:1787827138.753910:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827138.753917:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827138.753955:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790189440/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827138.753987:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827138.753990:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827138.753992:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827138.753996:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3775 00000100:00080000:0.0:1787827138.754000:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827138.754003:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827138.754012:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827138.754024:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790189568/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827138.754033:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827138.754036:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827138.754038:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827138.754041:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3775 00000100:00080000:0.0:1787827138.754045:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827138.754048:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827138.754052:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827138.754061:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790189696/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827138.754070:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827138.754072:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827138.754075:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827138.754078:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3775 00000100:00080000:0.0:1787827138.754082:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827138.754086:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827138.754092:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827138.754104:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790189824/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827138.754117:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827138.754121:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827138.754123:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827138.754127:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3775 00000100:00080000:0.0:1787827138.754132:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827138.754136:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827138.754141:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827138.754154:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790189952/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827138.754163:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827138.754166:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827138.754167:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827138.754172:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3775 00000100:00080000:0.0:1787827138.754177:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827138.754183:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827138.754188:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827138.754206:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790190080/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827138.754217:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827138.754230:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827138.754233:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827138.754238:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3775 00000100:00080000:0.0:1787827138.754245:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827138.754253:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827138.754261:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827138.754277:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790190208/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827138.754293:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827138.754301:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827138.754303:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827138.754308:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3775 00000100:00080000:0.0:1787827138.754312:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827138.754316:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827138.754322:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827138.754338:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790190336/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827138.754352:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827138.754361:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827138.754363:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827138.754371:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3775 00000100:00080000:0.0:1787827138.754377:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827138.754384:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827138.754391:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827138.754415:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790190464/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827139.265794:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827140.289863:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827141.314290:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827142.337790:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827143.361799:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827144.385787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827145.409835:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827146.433805:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827147.457803:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827148.481800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827149.505815:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827150.529776:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827151.553953:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827152.578546:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827153.601811:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827154.114133:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790192896:192.168.203.109@tcp:250:kworker.0 00000800:00000100:0.0:1787827154.114236:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827154.114306:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827154/real 1787827154] req@ffff9e680a2d52c0 x1874668790192896/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827154.114350:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790192896/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827154.114419:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827154.114434:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827154.114448:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790192896:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827154.114474:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790193024:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827154.114527:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827154.114542:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827154/real 1787827154] req@ffff9e680a2d61c0 x1874668790193024/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827154.114554:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790193024/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827154.114563:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827154.114567:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827154.114570:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790193024:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827154.114596:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790193152:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827154.114616:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827154.114627:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827154/real 1787827154] req@ffff9e680a2d7480 x1874668790193152/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827154.114638:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790193152/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827154.114646:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827154.114650:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827154.114653:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790193152:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827154.114660:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790193280:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827154.114678:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827154.114689:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827154/real 1787827154] req@ffff9e680a2d7c00 x1874668790193280/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827154.114733:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790193280/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827154.114743:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827154.114747:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827154.114750:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790193280:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827154.114758:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790193408:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827154.114779:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827154.114802:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827154/real 1787827154] req@ffff9e680a2d70c0 x1874668790193408/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827154.114815:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790193408/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827154.114822:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827154.114826:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827154.114829:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790193408:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827154.114836:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790193536:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827154.114855:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827154.114867:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827154/real 1787827154] req@ffff9e680a2d4780 x1874668790193536/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827154.114884:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790193536/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827154.116257:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827154.116265:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827154.116269:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790193536:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827154.116279:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790193664:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827154.116317:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827154.116334:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827154/real 1787827154] req@ffff9e680a2d4f00 x1874668790193664/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827154.116347:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790193664/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827154.116354:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827154.116357:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827154.116360:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790193664:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827154.116366:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790193792:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827154.116384:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827154.116393:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827154/real 1787827154] req@ffff9e680a2d43c0 x1874668790193792/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827154.116403:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790193792/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827154.116415:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827154.116418:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827154.116421:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790193792:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827154.116426:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790193920:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827154.116444:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827154.116454:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827154/real 1787827154] req@ffff9e680a2d4000 x1874668790193920/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827154.116465:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790193920/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827154.116472:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827154.116474:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827154.116476:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790193920:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827154.626200:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827155.649816:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827156.673805:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827157.697817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827158.722538:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827159.234941:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790194048:192.168.203.109@tcp:250:kworker.0 00000800:00000100:0.0:1787827159.234994:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827159.235026:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827159/real 1787827159] req@ffff9e680a2d5e00 x1874668790194048/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827159.235040:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790194048/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827159.235051:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827159.235054:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827159.235058:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790194048:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827159.235064:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790194176:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827159.235083:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827159.235094:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827159/real 1787827159] req@ffff9e680a2d6d00 x1874668790194176/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827159.235106:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790194176/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827159.235114:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827159.235117:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827159.235120:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790194176:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827159.235126:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790194304:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827159.235142:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827159.235151:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827159/real 1787827159] req@ffff9e680a2d5a40 x1874668790194304/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827159.235161:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790194304/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827159.235169:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827159.235172:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827159.235175:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790194304:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827159.235181:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790194432:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827159.235195:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827159.235204:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827159/real 1787827159] req@ffff9e680a2d4b40 x1874668790194432/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827159.235214:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790194432/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827159.235221:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827159.235224:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827159.235227:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790194432:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827159.235232:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790194560:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827159.235249:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827159.235258:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827159/real 1787827159] req@ffff9e680a2d6580 x1874668790194560/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827159.235269:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790194560/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827159.235276:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827159.235279:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827159.235282:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790194560:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827159.235287:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790194688:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827159.235303:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827159.235313:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827159/real 1787827159] req@ffff9e680a2d4000 x1874668790194688/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827159.235323:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790194688/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827159.235332:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827159.235335:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827159.235337:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790194688:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827159.235343:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790194816:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827159.235358:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827159.235369:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827159/real 1787827159] req@ffff9e680a2d43c0 x1874668790194816/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827159.235380:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790194816/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827159.235387:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827159.235389:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827159.235392:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790194816:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827159.235398:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790194944:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827159.235413:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827159.235421:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827159/real 1787827159] req@ffff9e680a2d4f00 x1874668790194944/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827159.235432:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790194944/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827159.235445:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827159.235447:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827159.235450:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790194944:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827159.235468:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790195072:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827159.235485:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827159.235494:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827159/real 1787827159] req@ffff9e680a2d4780 x1874668790195072/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827159.235504:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790195072/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827159.235512:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827159.235514:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827159.235517:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790195072:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827159.745812:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827160.769834:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827161.793785:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:0.0:1787827161.986976:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827162.817827:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827163.841770:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:1.0:1787827164.354260:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790195328:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827164.354285:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790195456:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827164.354306:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827164.354317:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827164/real 1787827164] req@ffff9e680a2d7480 x1874668790195456/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827164.354329:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790195456/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827164.354336:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827164.354339:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827164.354343:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790195456:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827164.354349:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790195584:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827164.354366:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827164.354376:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827164/real 1787827164] req@ffff9e680a2d61c0 x1874668790195584/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827164.354386:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790195584/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827164.354393:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827164.354396:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827164.354400:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790195584:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827164.354405:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790195712:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827164.354435:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827164.354445:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827164/real 1787827164] req@ffff9e680a2d52c0 x1874668790195712/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827164.354457:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790195712/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827164.354466:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827164.354469:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827164.354472:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790195712:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827164.354477:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790195840:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827164.354494:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827164.354504:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827164/real 1787827164] req@ffff9e680a2d4780 x1874668790195840/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827164.354515:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790195840/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827164.354523:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827164.354526:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827164.354529:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790195840:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827164.354534:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790195968:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827164.354552:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827164.354562:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827164/real 1787827164] req@ffff9e680a2d4f00 x1874668790195968/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827164.354572:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790195968/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827164.354585:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827164.354588:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827164.354591:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790195968:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827164.354596:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790196096:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827164.354614:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827164.354625:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827164/real 1787827164] req@ffff9e680a2d43c0 x1874668790196096/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827164.354635:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790196096/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827164.354642:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827164.354645:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827164.354648:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790196096:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827164.354653:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790196224:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827164.354670:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827164.354681:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827164/real 1787827164] req@ffff9e680a2d4000 x1874668790196224/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827164.354692:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790196224/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827164.354729:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827164.354732:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827164.354736:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790196224:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827164.865816:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827165.889807:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827166.913830:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827167.105897:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827167.937817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827168.961837:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:1.0:1787827169.473935:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790196352:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787827169.474036:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790196480:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827169.474071:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790196608:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827169.474098:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790196736:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827169.474126:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790196864:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827169.474153:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790196992:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827169.474242:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790197120:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827169.474278:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790197248:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827169.474328:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790197376:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827169.985789:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827171.009788:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827171.201996:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827172.033817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000400:00000100:0.0:1787827172.546054:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:0.0:1787827172.546117:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787827172.546138:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827172.546204:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:0.0:1787827172.546221:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827172.546231:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827172.546256:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827172.546268:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:1.0:1787827172.546269:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827169/real 1787827172] req@ffff9e680a2d4780 x1874668790197376/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787827172.546285:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827172.546303:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827172.546327:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:1.0:1787827172.546327:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790197376/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827172.546349:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827172.546355:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827172.546373:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790197376:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827172.546381:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827169/real 1787827172] req@ffff9e680a2d4f00 x1874668790197248/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827172.546397:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790197248/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827172.546405:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827172.546408:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827172.546412:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790197248:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827172.546418:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827169/real 1787827172] req@ffff9e680a2d43c0 x1874668790197120/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827172.546429:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790197120/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827172.546434:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827172.546436:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827172.546438:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790197120:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827172.546442:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827169/real 1787827172] req@ffff9e680a2d4000 x1874668790196992/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827172.546451:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790196992/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827172.546459:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827172.546462:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827172.546465:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790196992:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827172.546471:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827169/real 1787827172] req@ffff9e680a2d5e00 x1874668790196864/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827172.546482:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790196864/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827172.546489:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827172.546491:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827172.546494:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790196864:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827172.546500:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827169/real 1787827172] req@ffff9e680a2d4b40 x1874668790196480/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827172.546509:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790196480/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827172.546518:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827172.546521:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827172.546524:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790196480:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827172.546531:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827169/real 1787827172] req@ffff9e680a2d5a40 x1874668790196608/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827172.546542:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790196608/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827172.546549:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827172.546552:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827172.546555:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790196608:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827172.546561:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827169/real 1787827172] req@ffff9e680a2d6d00 x1874668790196736/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827172.546571:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790196736/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827172.546578:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827172.546581:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827172.546584:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790196736:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827172.546590:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827169/real 1787827172] req@ffff9e680a2d6580 x1874668790196352/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827172.546600:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790196352/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827172.546607:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827172.546610:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827172.546613:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790196352:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827173.057817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827174.081811:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:1.0:1787827174.593975:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790197504:192.168.203.109@tcp:250:kworker.0 00000800:00000100:1.0:1787827174.594031:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827174.594109:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827174/real 1787827174] req@ffff9e6804420b40 x1874668790197504/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827174.594135:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790197504/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827174.594152:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827174.594158:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827174.594166:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790197504:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787827174.594195:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790197632:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827174.594228:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827174.594246:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827174/real 1787827174] req@ffff9e6804422580 x1874668790197632/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827174.594269:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790197632/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827174.594285:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827174.594292:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827174.594300:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790197632:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827174.594317:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790197760:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827174.594355:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827174.594371:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827174/real 1787827174] req@ffff9e6804423480 x1874668790197760/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827174.594398:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790197760/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827174.594412:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827174.594419:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827174.594427:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790197760:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827174.594440:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790197888:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827174.594488:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827174.594506:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827174/real 1787827174] req@ffff9e68044230c0 x1874668790197888/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827174.594526:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790197888/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827174.594544:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827174.594552:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827174.594560:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790197888:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827174.594578:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790198016:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827174.594604:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827174.594628:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827174/real 1787827174] req@ffff9e6804422580 x1874668790198016/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827174.594643:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790198016/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827174.594651:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827174.594655:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827174.594658:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790198016:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827174.594664:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790198144:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827174.594692:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827174.594728:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827174/real 1787827174] req@ffff9e6804420b40 x1874668790198144/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827174.594747:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790198144/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827174.594755:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827174.594757:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827174.594760:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790198144:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827174.594765:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790198272:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827174.594782:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827174.594792:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827174/real 1787827174] req@ffff9e6804420f00 x1874668790198272/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827174.594803:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790198272/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827174.594810:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827174.594813:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827174.594816:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790198272:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827174.594822:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790198400:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827174.594839:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827174.594849:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827174/real 1787827174] req@ffff9e68044221c0 x1874668790198400/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827174.594859:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790198400/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827174.594873:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827174.594877:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827174.594879:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790198400:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827174.594895:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790198528:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827174.594912:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827174.594922:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827174/real 1787827174] req@ffff9e6804421680 x1874668790198528/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827174.594932:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790198528/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827174.594939:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827174.594943:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827174.594946:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790198528:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827175.105799:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827175.297880:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827176.129838:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827177.153808:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827178.177807:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827179.201798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827179.393963:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827179.713831:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827179.713845:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827179.713850:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827179.713862:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3816 00000100:00080000:0.0:1787827179.713876:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827179.713884:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827179.713928:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790198656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827179.713965:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827179.713969:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827179.713971:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827179.713975:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3816 00000100:00080000:0.0:1787827179.713978:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827179.713981:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827179.713992:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827179.714004:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790198784/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827179.714013:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827179.714018:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827179.714020:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827179.714020:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790198656:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787827179.714028:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3816 00000100:00080000:0.0:1787827179.714031:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827179.714034:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827179.714038:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827179.714047:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790198912/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827179.714056:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827179.714059:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827179.714061:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827179.714065:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3816 00000100:00080000:0.0:1787827179.714068:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827179.714073:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827179.714078:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827179.714092:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790199040/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827179.714104:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827179.714108:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827179.714112:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827179.714114:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827179.714119:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3816 00000100:00080000:0.0:1787827179.714122:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827179.714126:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827179.714131:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827179.714143:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790199168/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827179.714157:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827179.714157:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827179/real 1787827179] req@ffff9e6803e9bc00 x1874668790198656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827179.714165:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827179.714169:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827179.714175:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3816 00000100:00080000:0.0:1787827179.714178:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827179.714181:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790198656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827179.714185:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787827179.714192:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787827179.714193:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787827179.714201:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827179.714207:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790198656:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827179.714209:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790199296/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827179.714225:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827179.714229:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827179.714230:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827179.714234:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3816 00000100:00080000:0.0:1787827179.714237:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827179.714240:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827179.714245:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827179.714250:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790198784:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827179.714255:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790199424/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827179.714266:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827179.714270:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827179.714271:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827179.714275:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3816 00000800:00000100:1.0:1787827179.714279:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827179.714280:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827179.714287:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827179.714292:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827179.714294:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827179/real 1787827179] req@ffff9e6803e992c0 x1874668790198784/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827179.714304:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790198784/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827179.714324:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787827179.714326:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790199552/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827179.714329:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827179.714335:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790198784:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827179.714343:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827179.714350:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827179.714352:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827179.714357:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790198912:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827179.714358:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3816 00000100:00080000:0.0:1787827179.714364:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827179.714370:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827179.714375:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787827179.714387:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827179.714389:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790199680/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827179.714400:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827179/real 1787827179] req@ffff9e6803e9ad00 x1874668790198912/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827179.714409:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790198912/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827179.714414:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827179.714417:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827179.714420:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790198912:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827179.714424:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790199040:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827179.714438:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827179.714443:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827179/real 1787827179] req@ffff9e6803e9b480 x1874668790199040/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827179.714450:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790199040/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827179.714454:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827179.714456:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827179.714457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790199040:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827179.714461:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790199168:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827179.714470:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827179.714475:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827179/real 1787827179] req@ffff9e6803e98b40 x1874668790199168/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827179.714481:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790199168/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827179.714485:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827179.714487:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827179.714488:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790199168:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827179.714491:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790199296:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827179.714499:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827179.714504:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827179/real 1787827179] req@ffff9e6803e9a940 x1874668790199296/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827179.714514:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790199296/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827179.714520:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827179.714521:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827179.714523:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790199296:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827179.714534:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790199424:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827179.714543:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827179.714549:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827179/real 1787827179] req@ffff9e6803e9b840 x1874668790199424/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827179.714555:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790199424/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827179.714559:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827179.714561:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827179.714562:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790199424:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827179.714565:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790199552:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827179.714574:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827179.714579:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827179/real 1787827179] req@ffff9e6803e9a1c0 x1874668790199552/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827179.714585:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790199552/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827179.714593:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827179.714595:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827179.714597:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790199552:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827179.714600:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790199680:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827179.714608:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827179.714613:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827179/real 1787827179] req@ffff9e6803e9a580 x1874668790199680/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827179.714618:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790199680/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827179.714623:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827179.714625:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827179.714627:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790199680:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827180.225800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827181.249784:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827182.273780:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827183.297767:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827183.489906:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827184.321798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:1.0:1787827184.834022:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790199808:192.168.203.109@tcp:250:kworker.0 00000800:00000100:1.0:1787827184.834090:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827184.834135:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827184/real 1787827184] req@ffff9e6804422d00 x1874668790199808/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827184.834192:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790199808/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827184.834212:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827184.834223:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827184.834230:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790199808:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787827184.834251:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790199936:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827184.834305:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827184.834320:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827184/real 1787827184] req@ffff9e6804421e00 x1874668790199936/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827184.834354:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790199936/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827184.834368:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827184.834373:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827184.834382:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790199936:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827184.834402:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790200064:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827184.834421:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827184.834437:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827184/real 1787827184] req@ffff9e6804422940 x1874668790200064/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827184.834449:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790200064/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827184.834457:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827184.834460:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827184.834463:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790200064:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827184.834488:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790200192:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827184.834506:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827184.834516:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827184/real 1787827184] req@ffff9e6804423840 x1874668790200192/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827184.834525:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790200192/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827184.834532:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827184.834535:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827184.834537:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790200192:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827184.834543:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790200320:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827184.834556:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827184.834565:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827184/real 1787827184] req@ffff9e68044212c0 x1874668790200320/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827184.834574:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790200320/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827184.834581:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827184.834583:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827184.834586:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790200320:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827184.834591:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790200448:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827184.834605:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827184.834614:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827184/real 1787827184] req@ffff9e6804420780 x1874668790200448/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827184.834623:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790200448/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827184.834631:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827184.834634:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827184.834637:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790200448:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827184.834641:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790200576:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827184.834663:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827184.834673:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827184/real 1787827184] req@ffff9e6804421680 x1874668790200576/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827184.834683:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790200576/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827184.834689:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827184.834691:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827184.834693:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790200576:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827184.834723:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790200704:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827184.834738:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827184.834745:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827184/real 1787827184] req@ffff9e68044221c0 x1874668790200704/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827184.834752:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790200704/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827184.834756:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827184.834758:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827184.834760:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790200704:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827184.834763:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790200832:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827184.834773:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827184.834777:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827184/real 1787827184] req@ffff9e6804420f00 x1874668790200832/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827184.834783:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790200832/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827184.834787:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827184.834789:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827184.834791:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790200832:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827185.345789:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827186.369796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827187.393780:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827187.585943:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827188.417818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827189.441789:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:1.0:1787827189.954042:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790200960:192.168.203.109@tcp:250:kworker.0 00000800:00000100:1.0:1787827189.954129:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827189.954185:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827189/real 1787827189] req@ffff9e6804420b40 x1874668790200960/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827189.954218:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790200960/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827189.954264:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827189.954273:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827189.954281:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790200960:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787827189.954298:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790201088:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827189.954330:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827189.954357:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827189/real 1787827189] req@ffff9e6804422580 x1874668790201088/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827189.954373:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790201088/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827189.954385:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827189.954389:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827189.954395:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790201088:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827189.954401:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790201216:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827189.954414:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827189.954447:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827189/real 1787827189] req@ffff9e68044230c0 x1874668790201216/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827189.954457:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790201216/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827189.954464:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827189.954467:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827189.954470:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790201216:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827189.954474:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790201344:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827189.954504:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827189.954524:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827189/real 1787827189] req@ffff9e6804423480 x1874668790201344/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827189.954542:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790201344/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827189.954552:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827189.954555:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827189.954559:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790201344:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827189.954583:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790201472:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827189.954599:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827189.954607:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827189/real 1787827189] req@ffff9e6804420f00 x1874668790201472/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827189.954614:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790201472/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827189.954620:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827189.954622:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827189.954624:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790201472:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827189.954628:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790201600:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827189.954638:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827189.954644:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827189/real 1787827189] req@ffff9e68044221c0 x1874668790201600/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827189.954651:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790201600/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827189.954662:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827189.954671:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827189.954674:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790201600:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827189.954678:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790201728:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827189.954689:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827189.954729:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827189/real 1787827189] req@ffff9e6804421680 x1874668790201728/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827189.954738:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790201728/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827189.954746:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827189.954748:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827189.954750:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790201728:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827189.954756:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790201856:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827189.954768:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827189.954777:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827189/real 1787827189] req@ffff9e6804420780 x1874668790201856/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827189.954785:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790201856/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827189.954790:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827189.954792:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827189.954795:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790201856:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827189.954799:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790201984:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827189.954809:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827189.954815:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827189/real 1787827189] req@ffff9e68044212c0 x1874668790201984/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827189.954822:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790201984/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827189.954827:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827189.954829:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827189.954831:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790201984:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827190.465798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827191.489841:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827191.681932:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827192.513788:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827193.537783:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827194.561797:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00080000:0.0:1787827195.073830:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827195.073843:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827195.073848:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827195.073872:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3831 00000100:00080000:0.0:1787827195.073888:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827195.073897:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827195.073948:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790202112/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827195.074000:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827195.074005:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827195.074007:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827195.074011:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3831 00000100:00080000:0.0:1787827195.074015:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827195.074019:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827195.074031:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827195.074048:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790202240/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827195.074061:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827195.074065:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827195.074068:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827195.074073:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3831 00000100:00080000:0.0:1787827195.074077:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827195.074081:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827195.074086:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827195.074099:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790202368/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827195.074109:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790202112:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787827195.074117:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827195.074124:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827195.074126:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827195.074133:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3831 00000100:00080000:0.0:1787827195.074139:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827195.074145:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827195.074162:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827195.074175:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790202496/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827195.074186:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827195.074190:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827195.074192:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827195.074196:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3831 00000100:00080000:0.0:1787827195.074201:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827195.074225:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827195.074226:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827195.074242:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827195.074278:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790202624/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827195.074291:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827195.074293:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827195/real 1787827195] req@ffff9e6803e98f00 x1874668790202112/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827195.074302:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827195.074304:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827195.074318:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3831 00000100:00080000:0.0:1787827195.074328:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827195.074333:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790202112/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827195.074340:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787827195.074355:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787827195.074357:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787827195.074361:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827195.074370:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790202112:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787827195.074379:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790202240:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827195.074394:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790202752/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827195.074415:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827195.074428:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827195.074430:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827195/real 1787827195] req@ffff9e6803e99e00 x1874668790202240/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827195.074439:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827195.074446:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827195.074449:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790202240/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827195.074464:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3831 00000100:00080000:1.0:1787827195.074471:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827195.074474:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827195.074478:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787827195.074485:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827195.074486:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790202240:192.168.203.109@tcp:38:kworker.0 00000008:00100000:0.0:1787827195.074501:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827195.074518:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790202880/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827195.074522:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790202368:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827195.074546:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827195.074550:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827195.074552:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827195.074557:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3831 00000800:00000100:1.0:1787827195.074557:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827195.074573:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827195.074578:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827195/real 1787827195] req@ffff9e6803e99680 x1874668790202368/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827195.074584:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827195.074599:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827195.074608:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790202368/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827195.074615:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827195.074617:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827195.074619:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790202368:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827195.074626:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790202496:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827195.074632:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790203008/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827195.074644:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827195.074648:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827195.074655:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827195/real 1787827195] req@ffff9e6803e98780 x1874668790202496/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827195.074656:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827195.074660:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827195.074665:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3831 00000100:00080000:0.0:1787827195.074677:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827195.074691:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827195.074745:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827195.074760:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790203136/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827195.074782:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790202496/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827195.074797:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827195.074802:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827195.074808:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790202496:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827195.074820:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790202624:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827195.074837:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827195.074845:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827195/real 1787827195] req@ffff9e6803e9a580 x1874668790202624/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827195.074852:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790202624/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827195.074858:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827195.074860:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827195.074862:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790202624:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827195.074866:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790202752:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827195.074876:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827195.074882:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827195/real 1787827195] req@ffff9e6803e9a1c0 x1874668790202752/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827195.074889:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790202752/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827195.074900:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827195.074902:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827195.074903:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790202752:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827195.074915:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790202880:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827195.074925:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827195.074931:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827195/real 1787827195] req@ffff9e6803e9b840 x1874668790202880/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827195.074939:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790202880/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827195.074944:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827195.074946:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827195.074947:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790202880:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827195.074951:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790203008:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827195.074961:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827195.074966:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827195/real 1787827195] req@ffff9e6803e9a940 x1874668790203008/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827195.074973:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790203008/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827195.074978:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827195.074980:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827195.074982:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790203008:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827195.075002:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790203136:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827195.075016:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827195.075023:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827195/real 1787827195] req@ffff9e6803e98b40 x1874668790203136/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827195.075031:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790203136/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827195.075037:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827195.075039:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827195.075041:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790203136:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827195.585819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827195.777906:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827196.609787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827197.633755:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827198.657785:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827199.681782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827199.873858:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827200.193781:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827200.193789:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827200.193792:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827200.193801:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3836 00000100:00080000:0.0:1787827200.193812:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827200.193820:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827200.193858:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790203264/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827200.193882:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827200.193885:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827200.193886:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827200.193889:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3836 00000100:00080000:0.0:1787827200.193892:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827200.193895:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827200.193903:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827200.193912:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790203392/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827200.193920:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827200.193923:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827200.193932:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827200.193936:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3836 00000100:00080000:0.0:1787827200.193944:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827200.193951:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827200.193952:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790203264:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1787827200.193957:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827200.193973:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790203520/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827200.193985:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827200.193989:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827200.193992:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827200.193996:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3836 00000100:00080000:0.0:1787827200.194000:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827200.194004:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827200.194011:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827200.194021:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790203648/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827200.194027:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827200.194030:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827200.194034:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787827200.194035:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827200.194040:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3836 00000100:00080000:0.0:1787827200.194044:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827200.194047:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827200.194050:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827200.194060:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790203776/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827200.194066:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827200.194069:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827200.194070:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827200.194072:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3836 00000100:00080000:0.0:1787827200.194076:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827200.194079:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827200.194082:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827200.194084:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827200/real 1787827200] req@ffff9e6803e9b480 x1874668790203264/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827200.194092:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790203904/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827200.194112:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827200.194115:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827200.194117:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827200.194121:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3836 00000100:00100000:1.0:1787827200.194122:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790203264/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827200.194124:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827200.194129:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827200.194134:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827200.194143:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790204032/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827200.194147:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827200.194154:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827200.194158:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827200.194158:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787827200.194162:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827200.194184:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790203264:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787827200.194187:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3836 00000100:00080000:0.0:1787827200.194193:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827200.194197:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827200.194202:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790203392:192.168.203.109@tcp:38:kworker.0 00000008:00100000:0.0:1787827200.194203:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827200.194215:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790204160/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827200.194223:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827200.194225:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827200.194227:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827200.194230:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3836 00000100:00080000:0.0:1787827200.194233:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827200.194238:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1787827200.194241:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787827200.194243:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827200.194253:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790204288/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827200.194265:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827200/real 1787827200] req@ffff9e6803e9ad00 x1874668790203392/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827200.194283:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790203392/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827200.194293:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827200.194297:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827200.194301:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790203392:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827200.194326:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790203520:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827200.194349:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827200.194362:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827200/real 1787827200] req@ffff9e6803e992c0 x1874668790203520/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827200.194375:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790203520/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827200.194384:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827200.194387:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827200.194391:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790203520:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827200.194397:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790203648:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827200.194418:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827200.194429:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827200/real 1787827200] req@ffff9e6803e9bc00 x1874668790203648/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827200.194449:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790203648/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827200.194457:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827200.194461:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827200.194464:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790203648:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827200.194471:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790203776:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827200.194491:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827200.194502:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827200/real 1787827200] req@ffff9e6803e98b40 x1874668790203776/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827200.194514:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790203776/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827200.194522:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827200.194525:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827200.194529:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790203776:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827200.194534:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790203904:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827200.194552:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827200.194562:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827200/real 1787827200] req@ffff9e6803e9a940 x1874668790203904/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827200.194573:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790203904/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827200.194588:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827200.194591:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827200.194595:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790203904:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827200.194600:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790204032:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827200.194617:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827200.194626:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827200/real 1787827200] req@ffff9e6803e9b840 x1874668790204032/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827200.194636:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790204032/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827200.194645:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827200.194648:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827200.194651:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790204032:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827200.194657:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790204160:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827200.194675:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827200.194685:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827200/real 1787827200] req@ffff9e6803e9a1c0 x1874668790204160/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827200.194724:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790204160/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827200.194732:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827200.194735:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827200.194738:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790204160:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827200.194744:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790204288:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827200.194764:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827200.194774:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827200/real 1787827200] req@ffff9e6803e9a580 x1874668790204288/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827200.194786:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790204288/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827200.194793:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827200.194796:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827200.194799:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790204288:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827200.705777:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827201.729827:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827202.753746:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827203.777838:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827203.969952:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827204.801808:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00080000:0.0:1787827205.313776:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827205.313790:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827205.313796:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827205.313810:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3842 00000100:00080000:0.0:1787827205.313826:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827205.313837:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827205.313881:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790204416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827205.313909:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827205.313913:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827205.313915:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827205.313919:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3842 00000100:00080000:0.0:1787827205.313923:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827205.313928:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827205.313940:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827205.313949:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790204416:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827205.313957:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790204544/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827205.313969:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827205.313973:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827205.313975:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827205.313979:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3842 00000100:00080000:0.0:1787827205.313983:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827205.313987:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827205.313992:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827205.314005:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790204672/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827205.314014:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827205.314018:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787827205.314022:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827205.314023:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827205.314031:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3842 00000100:00080000:0.0:1787827205.314036:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827205.314042:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827205.314048:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827205.314060:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790204800/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827205.314067:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827205/real 1787827205] req@ffff9e6803e98780 x1874668790204416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827205.314075:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827205.314082:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827205.314085:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827205.314090:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3842 00000100:00100000:1.0:1787827205.314092:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790204416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827205.314095:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827205.314104:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787827205.314107:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787827205.314110:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787827205.314113:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827205.314126:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790204928/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827205.314127:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790204416:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787827205.314158:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827205.314162:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827205.314164:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827205.314169:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790204544:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827205.314173:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3842 00000100:00080000:0.0:1787827205.314182:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827205.314188:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827205.314193:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827205.314203:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827205.314208:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790205056/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827205.314217:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827205/real 1787827205] req@ffff9e6803e99680 x1874668790204544/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827205.314223:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827205.314229:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827205.314233:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827205.314233:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790204544/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827205.314243:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3842 00000100:00080000:1.0:1787827205.314249:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827205.314250:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827205.314257:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787827205.314260:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827205.314266:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827205.314271:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790204544:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827205.314277:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790205184/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827205.314281:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790204672:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827205.314296:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827205.314300:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827205.314302:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787827205.314304:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827205.314311:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3842 00000100:00080000:0.0:1787827205.314317:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827205.314318:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827205/real 1787827205] req@ffff9e6803e99e00 x1874668790204672/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827205.314327:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827205.314336:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827205.314339:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790204672/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827205.314349:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790205312/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827205.314361:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827205.314366:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827205.314369:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787827205.314372:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827205.314378:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827205.314378:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790204672:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827205.314387:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3842 00000100:00080000:0.0:1787827205.314391:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827205.314392:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790204800:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787827205.314398:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827205.314404:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827205.314417:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790205440/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827205.314420:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827205.314433:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827205/real 1787827205] req@ffff9e6803e98f00 x1874668790204800/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827205.314448:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790204800/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827205.314455:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827205.314458:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827205.314460:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790204800:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827205.314464:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790204928:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827205.314476:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827205.314482:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827205/real 1787827205] req@ffff9e6803e9a580 x1874668790204928/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827205.314489:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790204928/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827205.314494:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827205.314496:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827205.314498:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790204928:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827205.314509:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790205056:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827205.314520:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827205.314535:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827205/real 1787827205] req@ffff9e6803e9a1c0 x1874668790205056/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827205.314548:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790205056/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827205.314565:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827205.314569:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827205.314573:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790205056:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827205.314578:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790205184:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827205.314595:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827205.314603:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827205/real 1787827205] req@ffff9e6803e9b840 x1874668790205184/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827205.314610:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790205184/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827205.314616:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827205.314618:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827205.314620:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790205184:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827205.314624:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790205312:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827205.314633:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827205.314639:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827205/real 1787827205] req@ffff9e6803e9a940 x1874668790205312/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827205.314650:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790205312/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827205.314658:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827205.314662:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827205.314665:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790205312:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827205.314672:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790205440:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827205.314713:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827205.314724:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827205/real 1787827205] req@ffff9e6803e98b40 x1874668790205440/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827205.314733:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790205440/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827205.314738:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827205.314740:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827205.314742:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790205440:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827205.825817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827206.849798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827207.873754:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827208.065952:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827208.897826:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827209.921753:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:1.0:1787827210.434081:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790205568:192.168.203.109@tcp:250:kworker.0 00000800:00000100:1.0:1787827210.434169:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827210.434213:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827210/real 1787827210] req@ffff9e6804423840 x1874668790205568/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827210.434231:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790205568/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827210.434246:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827210.434257:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827210.434264:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790205568:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787827210.434291:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790205696:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827210.434323:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827210.434338:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827210/real 1787827210] req@ffff9e6804422940 x1874668790205696/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827210.434369:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790205696/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827210.434391:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827210.434399:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827210.434403:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790205696:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827210.434412:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790205824:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827210.434440:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827210.434451:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827210/real 1787827210] req@ffff9e6804421e00 x1874668790205824/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827210.434508:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790205824/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827210.434532:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827210.434540:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827210.434555:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790205824:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827210.434565:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790205952:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827210.434586:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827210.434596:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827210/real 1787827210] req@ffff9e6804422d00 x1874668790205952/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827210.434619:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790205952/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827210.434648:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827210.434660:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827210.434670:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790205952:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827210.434692:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790206080:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827210.434747:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827210.434764:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827210/real 1787827210] req@ffff9e6804423840 x1874668790206080/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827210.434812:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790206080/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827210.434836:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827210.434844:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827210.434852:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790206080:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827210.434870:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790206208:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827210.434902:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827210.434926:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827210/real 1787827210] req@ffff9e68044212c0 x1874668790206208/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827210.434995:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790206208/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827210.435017:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827210.435021:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827210.435024:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790206208:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827210.435029:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790206336:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827210.435046:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827210.435054:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827210/real 1787827210] req@ffff9e6804420780 x1874668790206336/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827210.435062:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790206336/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827210.435067:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827210.435069:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827210.435071:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790206336:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827210.435081:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790206464:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827210.435092:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827210.435119:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827210/real 1787827210] req@ffff9e6804421680 x1874668790206464/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827210.435128:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790206464/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827210.435133:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827210.435135:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827210.435138:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790206464:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827210.435142:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790206592:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827210.435157:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827210.435164:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827210/real 1787827210] req@ffff9e68044221c0 x1874668790206592/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827210.435171:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790206592/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827210.435176:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827210.435178:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827210.435180:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790206592:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827210.945834:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827211.969780:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827212.161915:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827212.993863:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827214.017824:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827215.041800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:1.0:1787827215.554092:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790206720:192.168.203.109@tcp:250:kworker.0 00000800:00000100:1.0:1787827215.554180:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827215.554232:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827215/real 1787827215] req@ffff9e680a2d52c0 x1874668790206720/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827215.554290:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790206720/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827215.554307:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827215.554323:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827215.554337:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790206720:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787827215.554363:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790206848:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827215.554406:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827215.554423:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827215/real 1787827215] req@ffff9e680a2d61c0 x1874668790206848/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827215.554441:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790206848/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827215.554452:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827215.554463:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827215.554467:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790206848:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827215.554475:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790206976:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827215.554498:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827215.554511:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827215/real 1787827215] req@ffff9e680a2d7480 x1874668790206976/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827215.554524:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790206976/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827215.554532:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827215.554536:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827215.554538:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790206976:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827215.554560:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790207104:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827215.554576:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827215.554583:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827215/real 1787827215] req@ffff9e680a2d7c00 x1874668790207104/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827215.554595:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790207104/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827215.554601:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827215.554603:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827215.554604:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790207104:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827215.554608:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790207232:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827215.554624:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827215.554643:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827215/real 1787827215] req@ffff9e680a2d70c0 x1874668790207232/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827215.554657:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790207232/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827215.554665:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827215.554668:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827215.554672:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790207232:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827215.554678:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790207360:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827215.554726:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827215.554738:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827215/real 1787827215] req@ffff9e680a2d6580 x1874668790207360/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827215.554750:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790207360/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827215.554761:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827215.554763:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827215.554765:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790207360:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827215.554770:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790207488:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827215.554785:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827215.554792:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827215/real 1787827215] req@ffff9e680a2d6d00 x1874668790207488/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827215.554799:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790207488/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827215.554804:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827215.554806:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827215.554808:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790207488:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827215.554812:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790207616:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827215.554822:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827215.554828:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827215/real 1787827215] req@ffff9e680a2d5a40 x1874668790207616/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827215.554835:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790207616/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827215.554840:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827215.554842:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827215.554844:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790207616:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827215.554847:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790207744:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827215.554857:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827215.554863:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827215/real 1787827215] req@ffff9e680a2d4b40 x1874668790207744/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827215.554869:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790207744/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827215.554874:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827215.554875:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827215.554877:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790207744:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827216.065793:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827216.257917:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827217.089815:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827218.113864:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827219.137795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827220.161782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827220.353950:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827220.673835:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827220.673847:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827220.673852:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827220.673863:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3857 00000100:00080000:0.0:1787827220.673875:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827220.673882:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827220.673928:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790207872/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827220.673957:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827220.673964:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827220.673965:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827220.673969:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3857 00000100:00080000:0.0:1787827220.673971:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827220.673975:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827220.673985:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827220.673996:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790208000/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827220.674008:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790207872:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787827220.674036:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827220.674039:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827220.674041:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827220.674046:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3857 00000100:00080000:0.0:1787827220.674049:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827220.674052:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827220.674056:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827220.674068:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790208128/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827220.674081:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827220.674084:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827220.674086:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827220.674088:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3857 00000100:00080000:0.0:1787827220.674091:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827220.674097:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827220.674098:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827220.674104:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827220.674113:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790208256/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827220.674122:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827220.674125:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827220.674126:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827220.674129:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3857 00000100:00080000:0.0:1787827220.674131:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827220.674134:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827220.674144:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827220.674150:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827220/real 1787827220] req@ffff9e6803e9bc00 x1874668790207872/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827220.674159:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790208384/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827220.674167:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827220.674169:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790207872/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827220.674170:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827220.674173:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827220.674181:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3857 00000100:00080000:0.0:1787827220.674186:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827220.674188:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827220.674190:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827220.674197:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787827220.674197:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827220.674204:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790207872:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827220.674208:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790208512/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827220.674216:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827220.674219:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827220.674220:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827220.674223:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3857 00000100:00080000:0.0:1787827220.674225:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827220.674227:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827220.674230:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827220.674234:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790208000:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827220.674239:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790208640/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827220.674248:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827220.674251:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827220.674253:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827220.674256:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3857 00000100:00080000:0.0:1787827220.674266:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827220.674267:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827220.674273:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827220.674282:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827220/real 1787827220] req@ffff9e6803e992c0 x1874668790208000/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787827220.674283:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827220.674298:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790208768/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827220.674299:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790208000/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827220.674328:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827220.674332:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827220.674333:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787827220.674338:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827220.674340:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827220.674346:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3857 00000100:00100000:1.0:1787827220.674348:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790208000:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827220.674353:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827220.674359:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790208128:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827220.674368:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827220.674376:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787827220.674383:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827220.674395:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790208896/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827220.674400:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827220/real 1787827220] req@ffff9e6803e9ad00 x1874668790208128/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827220.674427:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790208128/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827220.674437:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827220.674441:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827220.674443:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790208128:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827220.674449:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790208256:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827220.674466:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827220.674474:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827220/real 1787827220] req@ffff9e6803e9b480 x1874668790208256/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827220.674487:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790208256/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827220.674493:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827220.674495:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827220.674497:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790208256:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827220.674500:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790208384:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827220.674510:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827220.674516:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827220/real 1787827220] req@ffff9e6803e98b40 x1874668790208384/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827220.674523:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790208384/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827220.674529:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827220.674530:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827220.674532:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790208384:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827220.674535:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790208512:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827220.674545:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827220.674551:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827220/real 1787827220] req@ffff9e6803e9a940 x1874668790208512/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827220.674558:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790208512/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827220.674569:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827220.674571:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827220.674573:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790208512:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827220.674586:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790208640:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827220.674596:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827220.674602:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827220/real 1787827220] req@ffff9e6803e9b840 x1874668790208640/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827220.674609:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790208640/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827220.674615:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827220.674616:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827220.674618:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790208640:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827220.674622:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790208768:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827220.674631:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827220.674637:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827220/real 1787827220] req@ffff9e6803e9a1c0 x1874668790208768/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827220.674644:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790208768/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827220.674650:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827220.674651:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827220.674654:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790208768:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827220.674658:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790208896:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827220.674668:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827220.674673:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827220/real 1787827220] req@ffff9e6803e9a580 x1874668790208896/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827220.674680:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790208896/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827220.674685:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827220.674687:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827220.674689:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790208896:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827221.185826:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827222.209792:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827223.233795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827224.257860:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827224.449966:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827225.281806:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827225.793953:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790209024:192.168.203.109@tcp:250:kworker.0 00000800:00000100:0.0:1787827225.794022:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827225.794059:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827225/real 1787827225] req@ffff9e680a2d5e00 x1874668790209024/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827225.794080:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790209024/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827225.794097:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827225.794103:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827225.794111:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790209024:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827225.794139:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790209152:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827225.794164:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827225.794173:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827225/real 1787827225] req@ffff9e680a2d4000 x1874668790209152/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827225.794181:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790209152/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827225.794187:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827225.794190:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827225.794192:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790209152:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827225.794196:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790209280:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827225.794211:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827225.794223:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827225/real 1787827225] req@ffff9e680a2d43c0 x1874668790209280/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827225.794238:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790209280/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827225.794246:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827225.794249:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827225.794253:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790209280:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827225.794260:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790209408:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827225.794285:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827225.794299:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827225/real 1787827225] req@ffff9e680a2d4f00 x1874668790209408/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827225.794322:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790209408/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827225.794352:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827225.794356:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827225.794359:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790209408:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827225.794367:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790209536:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827225.794390:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827225.794410:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827225/real 1787827225] req@ffff9e680a2d4780 x1874668790209536/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827225.794428:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790209536/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827225.794449:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827225.794455:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827225.794462:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790209536:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827225.794484:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790209664:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827225.794514:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827225.794532:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827225/real 1787827225] req@ffff9e680a2d5e00 x1874668790209664/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827225.794550:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790209664/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827225.794585:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827225.794594:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827225.794602:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790209664:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827225.794628:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790209792:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827225.794651:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827225.794662:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827225/real 1787827225] req@ffff9e680a2d4b40 x1874668790209792/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827225.794672:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790209792/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827225.794681:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827225.794684:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827225.794687:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790209792:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827225.794693:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790209920:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827225.794735:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827225.794747:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827225/real 1787827225] req@ffff9e680a2d5a40 x1874668790209920/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827225.794756:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790209920/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827225.794762:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827225.794764:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827225.794767:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790209920:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827225.794776:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790210048:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827225.794787:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827225.794793:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827225/real 1787827225] req@ffff9e680a2d6d00 x1874668790210048/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827225.794800:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790210048/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827225.794806:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827225.794807:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827225.794809:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790210048:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827226.305811:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827227.329799:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827228.353804:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827228.545881:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827229.377780:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827230.401791:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827230.913969:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790210176:192.168.203.109@tcp:250:kworker.0 00000800:00000100:0.0:1787827230.914029:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827230.914074:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827230/real 1787827230] req@ffff9e6804420f00 x1874668790210176/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827230.914093:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790210176/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827230.914122:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827230.914128:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827230.914136:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790210176:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827230.914171:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790210304:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827230.914202:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827230.914224:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827230/real 1787827230] req@ffff9e6804423480 x1874668790210304/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827230.914248:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790210304/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827230.914281:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827230.914291:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827230.914300:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790210304:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827230.914312:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790210432:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827230.914337:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827230.914358:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827230/real 1787827230] req@ffff9e68044230c0 x1874668790210432/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827230.914388:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790210432/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827230.914403:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827230.914412:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827230.914417:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790210432:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827230.914427:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790210560:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827230.914449:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827230.914469:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827230/real 1787827230] req@ffff9e6804422580 x1874668790210560/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827230.914496:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790210560/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827230.914510:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827230.914517:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827230.914524:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790210560:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827230.914541:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790210688:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827230.914560:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827230.914568:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827230/real 1787827230] req@ffff9e6804420b40 x1874668790210688/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827230.914576:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790210688/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827230.914581:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827230.914583:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827230.914585:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790210688:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827230.914589:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790210816:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827230.914598:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827230.914604:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827230/real 1787827230] req@ffff9e6804420f00 x1874668790210816/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827230.914611:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790210816/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827230.914617:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827230.914619:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827230.914621:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790210816:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827230.914624:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790210944:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827230.914633:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827230.914639:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827230/real 1787827230] req@ffff9e68044221c0 x1874668790210944/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827230.914645:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790210944/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827230.914650:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827230.914652:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827230.914654:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790210944:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827230.914657:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790211072:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827230.914667:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827230.914672:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827230/real 1787827230] req@ffff9e6804421680 x1874668790211072/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827230.914684:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790211072/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827230.914692:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827230.914739:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827230.914743:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790211072:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827230.914747:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790211200:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827230.914765:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827230.914773:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827230/real 1787827230] req@ffff9e6804420780 x1874668790211200/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827230.914780:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790211200/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827230.914785:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827230.914787:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827230.914789:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790211200:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827231.425813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827232.449813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827232.641897:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827233.473752:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827234.497787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827235.521774:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827236.033942:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790211328:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827236.034048:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790211456:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827236.034100:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790211584:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827236.034129:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790211712:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827236.034158:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790211840:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827236.034186:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790211968:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827236.034203:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790212096:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827236.034231:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790212224:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827236.034244:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790212352:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827236.545800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827236.737855:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827237.569779:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827238.593788:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000400:00000100:0.0:1787827239.106086:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:0.0:1787827239.106132:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787827239.106149:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827239.106195:0:2387:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:0.0:1787827239.106208:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827239.106219:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:1.0:1787827239.106229:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827236/real 1787827239] req@ffff9e6804420f00 x1874668790212352/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787827239.106230:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827239.106244:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827239.106257:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:1.0:1787827239.106268:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790212352/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787827239.106272:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827239.106285:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00080000:1.0:1787827239.106289:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827239.106297:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827239.106301:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790212352:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827239.106310:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827236/real 1787827239] req@ffff9e6804422d00 x1874668790211584/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827239.106324:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790211584/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827239.106331:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827239.106335:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827239.106338:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790211584:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827239.106345:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827236/real 1787827239] req@ffff9e6804421e00 x1874668790211712/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827239.106356:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790211712/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827239.106364:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827239.106368:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827239.106371:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790211712:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827239.106377:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827236/real 1787827239] req@ffff9e6804422940 x1874668790211840/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827239.106390:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790211840/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827239.106399:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827239.106402:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827239.106405:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790211840:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827239.106412:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827236/real 1787827239] req@ffff9e6804420780 x1874668790211968/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827239.106425:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790211968/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827239.106434:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827239.106438:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827239.106441:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790211968:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827239.106447:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827236/real 1787827239] req@ffff9e6804421680 x1874668790212096/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827239.106457:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790212096/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827239.106465:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827239.106468:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827239.106472:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790212096:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827239.106478:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827236/real 1787827239] req@ffff9e68044212c0 x1874668790211328/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827239.106490:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790211328/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827239.106497:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827239.106501:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827239.106504:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790211328:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787827239.106510:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827236/real 1787827239] req@ffff9e6804423840 x1874668790211456/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827239.106523:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790211456/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827239.106531:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827239.106535:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827239.106538:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790211456:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827239.106568:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827236/real 1787827239] req@ffff9e68044221c0 x1874668790212224/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827239.106581:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790212224/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827239.106590:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827239.106594:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827239.106597:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790212224:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827239.617809:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827240.641794:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827240.833878:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827241.153811:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827241.153819:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827241.153822:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827241.153829:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3877 00000100:00080000:0.0:1787827241.153838:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827241.153843:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827241.153871:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790212480/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827241.153906:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827241.153909:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827241.153910:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827241.153913:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3877 00000100:00080000:0.0:1787827241.153916:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827241.153921:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827241.153928:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827241.153946:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790212480:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827241.153947:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790212608/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827241.153962:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827241.153965:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827241.153966:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827241.153969:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3877 00000100:00080000:0.0:1787827241.153972:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827241.153975:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827241.153979:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827241.153988:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790212736/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827241.153995:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827241.153997:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827241.153999:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827241.154002:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3877 00000100:00080000:0.0:1787827241.154005:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827241.154009:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827241.154011:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827241.154020:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827241.154030:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790212864/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827241.154040:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827241.154043:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827241.154045:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827241.154050:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3877 00000100:00100000:1.0:1787827241.154054:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827241/real 1787827241] req@ffff9e6803e98f00 x1874668790212480/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827241.154056:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827241.154066:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827241.154075:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827241.154085:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790212480/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827241.154090:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790212992/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827241.154110:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827241.154112:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827241.154119:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787827241.154120:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827241.154125:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827241.154130:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790212480:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787827241.154135:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3877 00000100:00080000:0.0:1787827241.154142:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827241.154148:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827241.154153:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827241.154162:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790212608:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827241.154168:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790213120/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827241.154181:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827241.154184:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827241.154186:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827241.154190:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3877 00000100:00080000:0.0:1787827241.154195:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827241.154199:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827241.154203:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827241.154211:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827241.154222:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827241/real 1787827241] req@ffff9e6803e99e00 x1874668790212608/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827241.154227:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790213248/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827241.154252:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827241.154257:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790212608/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827241.154258:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827241.154262:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827241.154271:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3877 00000100:00080000:1.0:1787827241.154277:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827241.154279:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827241.154286:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787827241.154288:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827241.154295:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790212608:192.168.203.109@tcp:38:kworker.0 00000008:00100000:0.0:1787827241.154298:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827241.154307:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790212736:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827241.154317:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790213376/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827241.154329:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827241.154333:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827241.154335:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827241.154338:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3877 00000100:00080000:0.0:1787827241.154343:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827241.154348:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827241.154352:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827241.154358:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827241.154369:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827241/real 1787827241] req@ffff9e6803e99680 x1874668790212736/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827241.154374:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790213504/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827241.154404:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790212736/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827241.154416:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827241.154424:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827241.154428:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790212736:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827241.154438:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790212864:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827241.154462:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827241.154475:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827241/real 1787827241] req@ffff9e6803e98780 x1874668790212864/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827241.154486:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790212864/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827241.154494:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827241.154497:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827241.154501:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790212864:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827241.154507:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790212992:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827241.154527:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827241.154539:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827241/real 1787827241] req@ffff9e6803e9a580 x1874668790212992/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827241.154550:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790212992/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827241.154557:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827241.154559:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827241.154563:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790212992:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827241.154586:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790213120:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827241.154606:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827241.154617:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827241/real 1787827241] req@ffff9e6803e9a1c0 x1874668790213120/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827241.154628:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790213120/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827241.154639:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827241.154642:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827241.154645:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790213120:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827241.154651:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790213248:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827241.154672:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827241.154683:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827241/real 1787827241] req@ffff9e6803e9b840 x1874668790213248/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827241.154721:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790213248/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827241.154731:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827241.154735:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827241.154738:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790213248:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827241.154745:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790213376:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827241.154768:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827241.154779:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827241/real 1787827241] req@ffff9e6803e9a940 x1874668790213376/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827241.154787:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790213376/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827241.154793:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827241.154795:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827241.154797:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790213376:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827241.154802:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790213504:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827241.154821:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827241.154834:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827241/real 1787827241] req@ffff9e6803e98b40 x1874668790213504/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827241.154845:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790213504/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827241.154852:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827241.154856:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827241.154860:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790213504:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827241.665782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827242.689798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827243.713791:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827244.737776:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827244.929915:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827245.761782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:1.0:1787827246.273968:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790213632:192.168.203.109@tcp:250:kworker.0 00000800:00000100:1.0:1787827246.274040:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827246.274093:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827246/real 1787827246] req@ffff9e680a2d6580 x1874668790213632/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827246.274126:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790213632/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827246.274150:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827246.274161:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827246.274168:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790213632:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787827246.274183:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790213760:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827246.274225:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827246.274241:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827246/real 1787827246] req@ffff9e680a2d70c0 x1874668790213760/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827246.274255:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790213760/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827246.274265:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827246.274269:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827246.274273:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790213760:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827246.274296:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790213888:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827246.274317:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827246.274327:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827246/real 1787827246] req@ffff9e680a2d7c00 x1874668790213888/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827246.274338:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790213888/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827246.274345:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827246.274348:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827246.274351:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790213888:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827246.274357:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790214016:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827246.274374:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827246.274383:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827246/real 1787827246] req@ffff9e680a2d7480 x1874668790214016/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827246.274400:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790214016/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827246.274407:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827246.274410:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827246.274413:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790214016:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827246.274418:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790214144:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827246.274433:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827246.274441:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827246/real 1787827246] req@ffff9e680a2d61c0 x1874668790214144/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827246.274451:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790214144/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827246.274458:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827246.274461:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827246.274464:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790214144:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827246.274469:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790214272:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827246.274482:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827246.274491:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827246/real 1787827246] req@ffff9e680a2d52c0 x1874668790214272/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827246.274502:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790214272/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827246.274519:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827246.274523:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827246.274526:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790214272:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827246.274533:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790214400:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827246.274559:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827246.274570:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827246/real 1787827246] req@ffff9e680a2d6d00 x1874668790214400/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827246.274583:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790214400/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827246.274592:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827246.274596:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827246.274599:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790214400:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827246.274606:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790214528:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827246.274626:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827246.274637:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827246/real 1787827246] req@ffff9e680a2d5a40 x1874668790214528/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827246.274650:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790214528/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827246.274659:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827246.274662:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827246.274666:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790214528:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827246.274672:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790214656:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827246.274693:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827246.274737:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827246/real 1787827246] req@ffff9e680a2d4b40 x1874668790214656/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827246.274749:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790214656/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827246.274758:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827246.274761:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827246.274765:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790214656:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827246.785769:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827247.809809:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827248.833777:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827249.025946:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827249.857884:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827250.881763:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00080000:1.0:1787827251.393854:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827251.393870:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827251.393876:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827251.393890:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3888 00000100:00080000:1.0:1787827251.393905:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827251.393915:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827251.393966:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790214784/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827251.393997:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827251.394000:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827251.394001:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827251.394004:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3888 00000100:00080000:1.0:1787827251.394008:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827251.394011:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827251.394022:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827251.394032:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790214912/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827251.394040:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827251.394043:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827251.394045:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827251.394048:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3888 00000100:00080000:1.0:1787827251.394050:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827251.394053:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827251.394056:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827251.394064:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790214784:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787827251.394068:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790215040/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827251.394089:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827251.394092:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827251.394094:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827251.394097:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3888 00000100:00080000:1.0:1787827251.394102:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827251.394105:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827251.394108:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827251.394118:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790215168/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827251.394130:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827251.394134:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827251.394135:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827251.394138:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3888 00000100:00080000:1.0:1787827251.394141:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787827251.394142:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827251.394149:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827251.394155:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827251.394169:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790215296/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827251.394176:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827251.394179:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827251.394181:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827251.394185:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3888 00000100:00100000:0.0:1787827251.394191:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827251/real 1787827251] req@ffff9e6831aa8000 x1874668790214784/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827251.394192:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827251.394206:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827251.394220:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827251.394237:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790215424/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827251.394238:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790214784/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827251.394300:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827251.394306:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827251.394308:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827251.394311:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827251.394316:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3888 00000100:00080000:0.0:1787827251.394324:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787827251.394326:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827251.394334:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827251.394335:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790214784:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787827251.394355:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827251.394361:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790214912:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827251.394368:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790215552/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827251.394377:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827251.394380:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827251.394393:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827251.394397:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3888 00000100:00080000:1.0:1787827251.394407:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827251.394412:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787827251.394414:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1787827251.394417:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827251.394430:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790215680/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827251.394437:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827251.394442:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827251/real 1787827251] req@ffff9e6831aab480 x1874668790214912/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827251.394443:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827251.394448:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827251.394455:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3888 00000100:00080000:1.0:1787827251.394465:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827251.394474:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827251.394478:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827251.394483:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790214912/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827251.394489:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790215808/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827251.394507:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827251.394522:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827251.394528:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790214912:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827251.394574:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790215040:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827251.394613:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827251.394625:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827251/real 1787827251] req@ffff9e6831aa9a40 x1874668790215040/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827251.394635:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790215040/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827251.394641:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827251.394644:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827251.394647:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790215040:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827251.394652:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790215168:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827251.394681:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827251.394687:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827251/real 1787827251] req@ffff9e6831aa8b40 x1874668790215168/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827251.394699:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790215168/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827251.394704:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827251.394706:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827251.394708:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790215168:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827251.394734:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790215296:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827251.394747:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827251.394754:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827251/real 1787827251] req@ffff9e6831aa8f00 x1874668790215296/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827251.394761:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790215296/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827251.394767:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827251.394769:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827251.394771:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790215296:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827251.394774:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790215424:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827251.394785:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827251.394791:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827251/real 1787827251] req@ffff9e6831aaa1c0 x1874668790215424/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827251.394798:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790215424/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827251.394808:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827251.394826:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827251.394829:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790215424:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827251.394833:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790215552:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827251.394849:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827251.394856:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827251/real 1787827251] req@ffff9e6831aa8780 x1874668790215552/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827251.394864:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790215552/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827251.394869:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827251.394871:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827251.394873:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790215552:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827251.394877:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790215680:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827251.394887:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827251.394892:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827251/real 1787827251] req@ffff9e6831aaa940 x1874668790215680/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827251.394899:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790215680/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827251.394904:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827251.394906:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827251.394908:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790215680:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827251.394912:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790215808:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827251.394922:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827251.394930:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827251/real 1787827251] req@ffff9e6831aaad00 x1874668790215808/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827251.394941:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790215808/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827251.394949:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827251.394953:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827251.394956:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790215808:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827251.905854:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827252.929815:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827253.121916:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827253.953791:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827254.977828:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827256.001806:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00080000:1.0:1787827256.513808:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827256.513816:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827256.513820:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827256.513827:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3893 00000100:00080000:1.0:1787827256.513837:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827256.513843:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827256.513880:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790215936/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827256.513910:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827256.513914:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827256.513916:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827256.513921:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3893 00000100:00080000:1.0:1787827256.513925:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827256.513930:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827256.513945:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827256.513961:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790216064/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827256.513971:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790215936:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827256.513972:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827256.513978:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827256.513981:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827256.513985:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3893 00000100:00080000:1.0:1787827256.513988:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827256.513991:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827256.513995:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827256.514005:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790216192/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827256.514012:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827256.514015:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827256.514016:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827256.514019:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3893 00000100:00080000:1.0:1787827256.514033:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827256.514035:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827256.514039:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787827256.514045:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827256.514049:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790216320/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827256.514058:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827256.514061:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827256.514062:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827256.514065:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3893 00000100:00080000:1.0:1787827256.514068:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827256.514071:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827256.514074:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827256.514088:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790216448/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827256.514094:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827256/real 1787827256] req@ffff9e6831aabc00 x1874668790215936/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827256.514097:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827256.514103:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827256.514105:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827256.514114:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3893 00000100:00080000:1.0:1787827256.514120:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827256.514123:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827256.514127:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827256.514130:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790215936/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827256.514136:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790216576/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827256.514154:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827256.514154:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827256.514157:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827256.514160:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827256.514164:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787827256.514165:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3893 00000100:00080000:1.0:1787827256.514169:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827256.514173:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827256.514178:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790215936:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787827256.514179:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827256.514189:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790216704/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827256.514196:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827256.514199:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827256.514200:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827256.514202:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3893 00000100:00080000:1.0:1787827256.514205:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827256.514207:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827256.514211:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827256.514214:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790216064:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827256.514221:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790216832/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827256.514229:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827256.514232:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827256.514233:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827256.514236:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3893 00000100:00080000:1.0:1787827256.514244:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827256.514251:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787827256.514252:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1787827256.514261:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827256.514274:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827256/real 1787827256] req@ffff9e6831aa9e00 x1874668790216064/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827256.514278:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790216960/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827256.514314:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790216064/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827256.514325:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827256.514329:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827256.514333:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790216064:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827256.514339:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790216192:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827256.514358:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827256.514368:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827256/real 1787827256] req@ffff9e6831aaa580 x1874668790216192/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827256.514380:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790216192/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827256.514389:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827256.514392:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827256.514395:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790216192:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827256.514401:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790216320:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827256.514420:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827256.514431:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827256/real 1787827256] req@ffff9e6831aa9680 x1874668790216320/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827256.514444:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790216320/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827256.514453:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827256.514456:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827256.514460:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790216320:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827256.514465:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790216448:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827256.514486:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827256.514498:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827256/real 1787827256] req@ffff9e6831aaad00 x1874668790216448/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827256.514509:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790216448/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827256.514517:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827256.514520:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827256.514523:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790216448:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827256.514529:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790216576:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827256.514547:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827256.514556:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827256/real 1787827256] req@ffff9e6831aaa940 x1874668790216576/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827256.514578:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790216576/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827256.514588:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827256.514591:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827256.514595:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790216576:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827256.514601:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790216704:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827256.514619:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827256.514629:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827256/real 1787827256] req@ffff9e6831aa8780 x1874668790216704/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827256.514643:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790216704/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827256.514651:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827256.514654:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827256.514658:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790216704:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827256.514679:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790216832:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827256.514721:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827256.514733:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827256/real 1787827256] req@ffff9e6831aaa1c0 x1874668790216832/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827256.514746:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790216832/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827256.514755:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827256.514758:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827256.514762:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790216832:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827256.514768:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790216960:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827256.514803:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827256.514814:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827256/real 1787827256] req@ffff9e6831aa8f00 x1874668790216960/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827256.514827:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790216960/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827256.514835:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827256.514839:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827256.514842:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790216960:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827257.025758:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827257.217852:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827258.049802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827259.073786:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827260.097819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827261.121796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827261.313924:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827261.633825:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827261.633836:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827261.633841:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827261.633851:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3898 00000100:00080000:1.0:1787827261.633863:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827261.633871:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827261.633920:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790217088/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827261.633969:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827261.633974:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827261.633977:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827261.633983:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3898 00000100:00080000:1.0:1787827261.633988:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827261.633993:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827261.634006:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827261.634026:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790217216/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827261.634037:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827261.634042:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827261.634045:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827261.634052:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3898 00000100:00080000:1.0:1787827261.634058:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827261.634065:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827261.634071:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827261.634083:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790217088:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787827261.634084:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790217344/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827261.634100:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827261.634103:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827261.634105:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827261.634108:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3898 00000100:00080000:1.0:1787827261.634111:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827261.634114:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827261.634117:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827261.634126:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790217472/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827261.634133:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827261.634136:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827261.634137:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827261.634140:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3898 00000100:00080000:1.0:1787827261.634161:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827261.634164:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827261.634167:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787827261.634177:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827261.634178:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790217600/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827261.634187:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827261.634190:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827261.634192:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827261.634195:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3898 00000100:00080000:1.0:1787827261.634197:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827261.634200:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827261.634203:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827261.634211:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790217728/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827261.634217:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827261.634219:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827261.634225:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827261.634229:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3898 00000100:00080000:1.0:1787827261.634232:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827261.634237:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827261/real 1787827261] req@ffff9e6831aa8b40 x1874668790217088/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827261.634237:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827261.634243:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827261.634256:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790217856/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827261.634263:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827261.634267:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787827261.634269:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790217088/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827261.634269:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827261.634277:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3898 00000100:00080000:1.0:1787827261.634281:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827261.634287:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827261.634292:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787827261.634296:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787827261.634301:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790217984/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827261.634307:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787827261.634311:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827261.634315:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787827261.634316:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790217088:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827261.634317:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827261.634325:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3898 00000100:00080000:1.0:1787827261.634327:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827261.634332:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827261.634334:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790217216:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787827261.634336:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827261.634347:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790218112/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827261.634369:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827261.634386:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827261/real 1787827261] req@ffff9e6831aa9a40 x1874668790217216/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827261.634400:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790217216/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827261.634410:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827261.634420:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827261.634425:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790217216:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827261.634455:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790217344:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827261.634480:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827261.634495:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827261/real 1787827261] req@ffff9e6831aab480 x1874668790217344/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827261.634512:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790217344/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827261.634524:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827261.634529:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827261.634538:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790217344:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827261.634548:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790217472:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827261.634568:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827261.634580:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827261/real 1787827261] req@ffff9e6831aa8000 x1874668790217472/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827261.634592:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790217472/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827261.634601:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827261.634604:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827261.634608:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790217472:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827261.634614:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790217600:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827261.634631:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827261.634640:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827261/real 1787827261] req@ffff9e6831aa8f00 x1874668790217600/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827261.634652:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790217600/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827261.634660:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827261.634663:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827261.634667:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790217600:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827261.634672:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790217728:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827261.634688:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827261.634730:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827261/real 1787827261] req@ffff9e6831aaa1c0 x1874668790217728/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827261.634761:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790217728/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827261.634773:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827261.634776:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827261.634780:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790217728:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827261.634786:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790217856:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827261.634806:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827261.634817:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827261/real 1787827261] req@ffff9e6831aa8780 x1874668790217856/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827261.634829:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790217856/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827261.634837:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827261.634841:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827261.634844:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790217856:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827261.634850:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790217984:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827261.634866:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827261.634876:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827261/real 1787827261] req@ffff9e6831aaa940 x1874668790217984/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827261.634887:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790217984/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827261.634896:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827261.634899:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827261.634903:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790217984:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827261.634908:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790218112:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827261.634928:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827261.634940:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827261/real 1787827261] req@ffff9e6831aaad00 x1874668790218112/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827261.634952:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790218112/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827261.634960:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827261.634963:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827261.634966:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790218112:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827262.145793:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827263.169791:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827264.193839:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827265.217781:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827265.409898:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827266.241753:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00080000:0.0:1787827266.753787:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827266.753795:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827266.753798:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827266.753806:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3903 00000100:00080000:0.0:1787827266.753814:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827266.753820:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827266.753851:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790218240/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827266.753875:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827266.753878:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827266.753879:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827266.753882:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3903 00000100:00080000:0.0:1787827266.753885:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827266.753888:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827266.753897:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827266.753908:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790218368/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827266.753919:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827266.753921:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827266.753922:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827266.753925:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3903 00000100:00080000:0.0:1787827266.753926:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827266.753928:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827266.753931:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827266.753939:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790218496/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827266.753962:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827266.753966:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827266.753977:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827266.753981:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3903 00000100:00080000:0.0:1787827266.753986:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827266.753990:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827266.753997:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827266.754013:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790218624/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827266.754022:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827266.754025:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827266.754026:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827266.754030:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3903 00000100:00080000:0.0:1787827266.754033:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827266.754036:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827266.754040:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827266.754054:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790218752/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827266.754067:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827266.754093:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827266.754097:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827266.754105:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3903 00000100:00080000:0.0:1787827266.754108:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827266.754114:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827266.754119:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827266.754130:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790218880/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827266.754137:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827266.754139:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827266.754141:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827266.754168:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3903 00000100:00080000:0.0:1787827266.754172:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827266.754180:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827266.754186:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827266.754201:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790219008/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827266.754210:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827266.754215:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827266.754219:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827266.754224:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3903 00000100:00080000:0.0:1787827266.754230:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827266.754233:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827266.754237:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827266.754249:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790219136/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827266.754284:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827266.754289:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827266.754292:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827266.754299:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3903 00000100:00080000:0.0:1787827266.754305:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827266.754309:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827266.754316:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827266.754331:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790219264/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827267.265788:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827268.289787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827269.313790:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827269.505913:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827270.337740:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827271.361800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827271.874063:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790219392:192.168.203.109@tcp:250:kworker.0 00000800:00000100:0.0:1787827271.874140:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827271.874184:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827271/real 1787827271] req@ffff9e6804420b40 x1874668790219392/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827271.874273:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790219392/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827271.874305:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827271.874314:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827271.874326:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790219392:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827271.874371:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790219520:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827271.874418:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827271.874442:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827271/real 1787827271] req@ffff9e6804422580 x1874668790219520/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827271.874494:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790219520/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827271.874553:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827271.874569:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827271.874586:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790219520:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827271.874598:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790219648:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827271.874623:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827271.874633:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827271/real 1787827271] req@ffff9e68044230c0 x1874668790219648/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827271.874641:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790219648/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827271.874647:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827271.874649:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827271.874651:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790219648:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827271.874655:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790219776:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827271.874666:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827271.874686:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827271/real 1787827271] req@ffff9e6804423480 x1874668790219776/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827271.874693:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790219776/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827271.874717:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827271.874719:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827271.874721:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790219776:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827271.874726:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790219904:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827271.874739:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827271.874746:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827271/real 1787827271] req@ffff9e68044221c0 x1874668790219904/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827271.874753:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790219904/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827271.874759:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827271.874761:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827271.874763:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790219904:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827271.874776:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790220032:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827271.874787:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827271.874793:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827271/real 1787827271] req@ffff9e6804423840 x1874668790220032/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827271.874800:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790220032/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827271.874806:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827271.874808:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827271.874810:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790220032:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827271.874813:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790220160:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827271.874823:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827271.874829:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827271/real 1787827271] req@ffff9e68044212c0 x1874668790220160/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827271.874840:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790220160/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827271.874845:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827271.874847:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827271.874849:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790220160:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827271.874852:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790220288:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827271.874861:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827271.874867:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827271/real 1787827271] req@ffff9e6804421680 x1874668790220288/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827271.874873:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790220288/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827271.874878:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827271.874880:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827271.874881:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790220288:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827271.874885:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790220416:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827271.874895:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827271.874900:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827271/real 1787827271] req@ffff9e6804420780 x1874668790220416/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827271.874906:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790220416/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827271.874915:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827271.874917:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827271.874919:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790220416:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827272.385813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827273.409763:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827273.601954:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827274.433772:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827275.457778:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:0.0:1787827276.994037:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790220544:192.168.203.109@tcp:250:kworker.0 00000800:00000100:0.0:1787827276.994136:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827276.994198:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827276/real 1787827276] req@ffff9e6804422940 x1874668790220544/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827276.994243:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790220544/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827276.994293:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827276.994309:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827276.994325:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790220544:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827276.994365:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790220672:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827276.994430:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827276.994477:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827276/real 1787827276] req@ffff9e6804421e00 x1874668790220672/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827276.994508:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790220672/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827276.994569:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827276.994582:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827276.994594:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790220672:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827276.994612:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790220800:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827276.994688:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827276.994704:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827276/real 1787827276] req@ffff9e6804422d00 x1874668790220800/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827276.994737:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790220800/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827276.994744:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827276.994747:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827276.994749:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790220800:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827276.994755:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790220928:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827276.994772:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827276.994783:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827276/real 1787827276] req@ffff9e6804420f00 x1874668790220928/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827276.994796:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790220928/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827276.994805:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827276.994809:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827276.994813:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790220928:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827276.994821:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790221056:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827276.994843:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827276.994855:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827276/real 1787827276] req@ffff9e6804420780 x1874668790221056/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827276.994869:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790221056/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827276.994879:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827276.994883:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827276.994886:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790221056:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827276.994893:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790221184:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827276.994909:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827276.994917:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827276/real 1787827276] req@ffff9e6804421680 x1874668790221184/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827276.994926:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790221184/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827276.994933:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827276.994935:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827276.994937:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790221184:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827276.994957:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790221312:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827276.994998:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827276.995013:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827276/real 1787827276] req@ffff9e68044212c0 x1874668790221312/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827276.995046:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790221312/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827276.995059:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827276.995063:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827276.995068:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790221312:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827276.995077:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790221440:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827276.995118:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827276.995131:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827276/real 1787827276] req@ffff9e6804423840 x1874668790221440/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827276.995143:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790221440/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827276.995150:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827276.995153:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827276.995156:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790221440:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827276.995180:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790221568:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827276.995205:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827276.995219:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827276/real 1787827276] req@ffff9e68044221c0 x1874668790221568/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827276.995249:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790221568/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827276.995258:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827276.995260:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827276.995263:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790221568:192.168.203.109@tcp:8:kworker.0 00010000:00010000:1.0:1787827277.697841:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827281.793860:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827282.113949:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790221696:192.168.203.109@tcp:250:kworker.0 00000800:00000100:0.0:1787827282.114011:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827282.114048:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827282/real 1787827282] req@ffff9e680a2d5e00 x1874668790221696/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827282.114064:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790221696/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827282.114075:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827282.114079:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827282.114083:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790221696:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827282.114105:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790221824:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827282.114127:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827282.114138:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827282/real 1787827282] req@ffff9e680a2d4780 x1874668790221824/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827282.114151:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790221824/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827282.114161:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827282.114164:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827282.114168:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790221824:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827282.114174:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790221952:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827282.114190:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827282.114200:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827282/real 1787827282] req@ffff9e680a2d4f00 x1874668790221952/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827282.114212:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790221952/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827282.114221:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827282.114225:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827282.114228:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790221952:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827282.114233:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790222080:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827282.114252:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827282.114263:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827282/real 1787827282] req@ffff9e680a2d43c0 x1874668790222080/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827282.114276:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790222080/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827282.114282:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827282.114284:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827282.114286:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790222080:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827282.114290:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790222208:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827282.114304:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827282.114311:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827282/real 1787827282] req@ffff9e680a2d4000 x1874668790222208/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827282.114318:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790222208/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827282.114327:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827282.114330:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827282.114333:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790222208:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827282.114351:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790222336:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827282.114371:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827282.114382:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827282/real 1787827282] req@ffff9e680a2d4b40 x1874668790222336/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827282.114394:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790222336/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827282.114404:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827282.114407:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827282.114411:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790222336:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827282.114417:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790222464:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827282.114437:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827282.114446:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827282/real 1787827282] req@ffff9e680a2d5a40 x1874668790222464/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827282.114454:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790222464/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827282.114464:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827282.114466:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827282.114468:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790222464:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827282.114472:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790222592:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827282.114483:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827282.114489:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827282/real 1787827282] req@ffff9e680a2d6d00 x1874668790222592/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827282.114496:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790222592/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827282.114501:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827282.114502:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827282.114504:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790222592:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827282.114508:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790222720:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827282.114517:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827282.114523:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827282/real 1787827282] req@ffff9e680a2d52c0 x1874668790222720/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827282.114530:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790222720/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827282.114534:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827282.114536:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827282.114538:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790222720:192.168.203.109@tcp:8:kworker.0 00010000:00010000:1.0:1787827285.889842:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827287.233795:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827287.233805:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827287.233810:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827287.233820:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3923 00000100:00080000:0.0:1787827287.233831:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827287.233839:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827287.233876:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790222848/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827287.233923:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827287.233927:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827287.233929:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827287.233933:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3923 00000100:00080000:0.0:1787827287.233937:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827287.233941:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827287.233950:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827287.233963:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790222976/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827287.233983:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827287.233987:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827287.233989:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827287.233994:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3923 00000100:00080000:0.0:1787827287.233998:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827287.234003:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827287.234008:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827287.234019:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790223104/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827287.234031:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827287.234037:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827287.234040:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827287.234045:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3923 00000100:00080000:0.0:1787827287.234051:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827287.234056:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827287.234062:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827287.234076:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790223232/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827287.234091:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827287.234100:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827287.234103:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827287.234109:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3923 00000100:00080000:0.0:1787827287.234118:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827287.234127:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827287.234132:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827287.234148:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790223360/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827287.234161:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827287.234166:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827287.234169:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827287.234174:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3923 00000100:00080000:0.0:1787827287.234179:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827287.234189:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827287.234196:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827287.234209:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790223488/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827287.234229:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827287.234236:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827287.234238:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827287.234242:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3923 00000100:00080000:0.0:1787827287.234248:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827287.234253:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827287.234259:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827287.234272:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790223616/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827287.234280:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827287.234283:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827287.234285:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827287.234288:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3923 00000100:00080000:0.0:1787827287.234292:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827287.234300:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827287.234307:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827287.234329:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790223744/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827287.234364:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827287.234373:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827287.234379:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827287.234388:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3923 00000100:00080000:0.0:1787827287.234393:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827287.234399:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827287.234423:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827287.234440:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790223872/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827289.985848:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827292.353768:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827292.353776:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827292.353780:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827292.353788:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3929 00000100:00080000:0.0:1787827292.353797:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827292.353802:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827292.353832:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790224000/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827292.353855:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827292.353858:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827292.353859:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827292.353862:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3929 00000100:00080000:0.0:1787827292.353865:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827292.353873:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827292.353882:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827292.353892:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790224128/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827292.353902:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827292.353905:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827292.353906:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827292.353908:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3929 00000100:00080000:0.0:1787827292.353912:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827292.353914:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827292.353917:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827292.353923:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790224256/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827292.353933:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827292.353935:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827292.353936:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827292.353939:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3929 00000100:00080000:0.0:1787827292.353942:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827292.353946:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827292.353953:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827292.353962:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790224384/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827292.353982:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827292.353984:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827292.353985:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827292.353988:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3929 00000100:00080000:0.0:1787827292.353990:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827292.353992:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827292.353996:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827292.354004:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790224512/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827292.354010:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827292.354014:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827292.354015:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827292.354018:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3929 00000100:00080000:0.0:1787827292.354020:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827292.354024:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827292.354028:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827292.354036:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790224640/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827292.354042:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827292.354044:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827292.354045:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827292.354048:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3929 00000100:00080000:0.0:1787827292.354050:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827292.354058:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827292.354066:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827292.354074:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790224768/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827292.354081:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827292.354084:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827292.354086:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827292.354089:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3929 00000100:00080000:0.0:1787827292.354091:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827292.354094:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827292.354097:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827292.354114:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790224896/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827292.354166:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827292.354169:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827292.354171:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827292.354179:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3929 00000100:00080000:0.0:1787827292.354189:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827292.354199:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827292.354204:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827292.354219:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790225024/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827294.081847:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827297.473938:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827297.473952:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827297.473959:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827297.473976:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3934 00000100:00080000:1.0:1787827297.474005:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827297.474016:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827297.474062:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790225152/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827297.474115:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827297.474123:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827297.474128:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827297.474138:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3934 00000100:00080000:1.0:1787827297.474148:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827297.474157:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827297.474172:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827297.474196:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790225280/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827297.474251:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827297.474258:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827297.474264:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827297.474275:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3934 00000100:00080000:1.0:1787827297.474285:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827297.474297:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827297.474308:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827297.474339:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790225408/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827297.474358:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827297.474364:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827297.474369:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827297.474381:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3934 00000100:00080000:1.0:1787827297.474396:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827297.474404:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827297.474413:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827297.474434:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790225536/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827297.474452:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827297.474456:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827297.474458:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827297.474463:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3934 00000100:00080000:1.0:1787827297.474469:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827297.474474:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827297.474481:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827297.474535:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790225664/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827297.474557:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827297.474565:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827297.474573:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827297.474590:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3934 00000100:00080000:1.0:1787827297.474598:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827297.474605:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827297.474613:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827297.474634:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790225792/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827297.474649:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827297.474653:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827297.474655:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827297.474660:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3934 00000100:00080000:1.0:1787827297.474665:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827297.474669:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827297.474674:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827297.474687:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790225920/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827297.474755:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827297.474762:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827297.474766:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827297.474775:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3934 00000100:00080000:1.0:1787827297.474782:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827297.474788:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827297.474797:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827297.474821:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790226048/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827297.474840:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827297.474858:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827297.474864:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827297.474872:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3934 00000100:00080000:1.0:1787827297.474884:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827297.474898:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827297.474915:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827297.474938:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790226176/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827298.177936:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827302.273939:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787827305.665914:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:0.0:1787827305.665951:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787827305.665968:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827305.666020:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:0.0:1787827305.666033:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827305.666044:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827305.666056:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827305.666069:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827305.666080:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827305.666093:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827305.666107:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00010000:00010000:1.0:1787827306.369923:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827307.713871:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827307.713882:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827307.713886:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827307.713897:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 3944 00000100:00080000:0.0:1787827307.713907:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827307.713915:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827307.713948:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790227456/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827307.713979:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827307.713984:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827307.713986:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827307.713991:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3944 00000100:00080000:0.0:1787827307.713995:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827307.714000:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827307.714010:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827307.714024:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790227584/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827307.714036:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827307.714041:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827307.714046:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827307.714052:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3944 00000100:00080000:0.0:1787827307.714056:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827307.714060:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827307.714065:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827307.714078:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790227712/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827307.714088:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827307.714091:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827307.714093:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827307.714097:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3944 00000100:00080000:0.0:1787827307.714110:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827307.714115:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827307.714121:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827307.714136:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790227840/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827307.714147:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827307.714150:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827307.714152:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827307.714156:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 3944 00000100:00080000:0.0:1787827307.714161:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827307.714168:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827307.714176:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827307.714195:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790227968/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827307.714220:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827307.714225:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827307.714229:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827307.714238:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3944 00000100:00080000:0.0:1787827307.714245:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827307.714254:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827307.714259:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827307.714272:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790228096/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827307.714286:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827307.714290:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827307.714293:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827307.714297:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3944 00000100:00080000:0.0:1787827307.714303:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827307.714308:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827307.714314:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827307.714330:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790228224/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827307.714356:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827307.714363:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827307.714368:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827307.714377:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3944 00000100:00080000:0.0:1787827307.714383:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827307.714392:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827307.714399:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827307.714435:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790228352/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827307.714447:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827307.714451:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827307.714453:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827307.714462:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 3944 00000100:00080000:0.0:1787827307.714469:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827307.714474:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827307.714482:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827307.714500:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790228480/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827310.465845:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827314.561895:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827318.657849:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827322.753894:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827326.849865:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827330.946230:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827333.313907:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790233216:192.168.203.109@tcp:250:kworker.0 00000800:00000100:0.0:1787827333.313996:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827333.314052:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827333/real 1787827333] req@ffff9e680a2d61c0 x1874668790233216/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827333.314095:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790233216/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827333.314106:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827333.314111:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827333.314114:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790233216:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827333.314137:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790233344:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827333.314167:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827333.314178:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827333/real 1787827333] req@ffff9e680a2d4000 x1874668790233344/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827333.314188:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790233344/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827333.314195:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827333.314197:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827333.314200:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790233344:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827333.314204:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790233472:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827333.314216:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827333.314224:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827333/real 1787827333] req@ffff9e680a2d43c0 x1874668790233472/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827333.314235:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790233472/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827333.314267:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827333.314273:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827333.314283:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790233472:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827333.314296:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790233600:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827333.314326:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827333.314350:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827333/real 1787827333] req@ffff9e680a2d4f00 x1874668790233600/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827333.314376:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790233600/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827333.314399:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827333.314410:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827333.314420:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790233600:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827333.314438:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790233728:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827333.314459:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827333.314472:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827333/real 1787827333] req@ffff9e680a2d4780 x1874668790233728/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827333.314484:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790233728/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827333.314493:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827333.314496:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827333.314501:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790233728:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827333.314521:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790233856:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827333.314537:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827333.314545:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827333/real 1787827333] req@ffff9e680a2d5e00 x1874668790233856/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827333.314553:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790233856/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827333.314560:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827333.314562:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827333.314564:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790233856:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827333.314568:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790233984:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827333.314578:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827333.314584:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827333/real 1787827333] req@ffff9e680a2d61c0 x1874668790233984/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827333.314591:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790233984/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827333.314596:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827333.314598:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827333.314600:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790233984:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827333.314603:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790234112:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827333.314612:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827333.314618:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827333/real 1787827333] req@ffff9e680a2d70c0 x1874668790234112/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827333.314625:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790234112/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827333.314631:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827333.314632:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827333.314634:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790234112:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827333.314637:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790234240:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827333.314646:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827333.314652:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827333/real 1787827333] req@ffff9e680a2d7c00 x1874668790234240/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827333.314659:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790234240/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827333.314664:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827333.314665:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827333.314668:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790234240:192.168.203.109@tcp:8:kworker.0 00010000:00010000:1.0:1787827335.042395:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827338.433933:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790234368:192.168.203.109@tcp:250:kworker.0 00000800:00000100:0.0:1787827338.433993:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827338.434035:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827338/real 1787827338] req@ffff9e680a2d7480 x1874668790234368/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827338.434063:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790234368/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827338.434080:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827338.434089:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827338.434098:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790234368:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827338.434124:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790234496:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827338.434153:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827338.434178:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827338/real 1787827338] req@ffff9e680a2d4b40 x1874668790234496/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827338.434197:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790234496/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827338.434221:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827338.434230:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827338.434236:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790234496:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827338.434246:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790234624:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827338.434270:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827338.434283:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827338/real 1787827338] req@ffff9e680a2d52c0 x1874668790234624/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827338.434310:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790234624/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827338.434330:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827338.434338:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827338.434343:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790234624:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827338.434350:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790234752:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827338.434366:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827338.434375:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827338/real 1787827338] req@ffff9e680a2d6580 x1874668790234752/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827338.434385:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790234752/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827338.434391:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827338.434393:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827338.434396:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790234752:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827338.434406:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790234880:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827338.434416:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827338.434421:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827338/real 1787827338] req@ffff9e680a2d6d00 x1874668790234880/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827338.434433:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790234880/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827338.434442:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827338.434445:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827338.434448:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790234880:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827338.434454:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790235008:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827338.434474:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827338.434486:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827338/real 1787827338] req@ffff9e680a2d5a40 x1874668790235008/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827338.434494:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790235008/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827338.434500:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827338.434503:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827338.434505:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790235008:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827338.434508:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790235136:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827338.434532:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827338.434539:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827338/real 1787827338] req@ffff9e680a2d7480 x1874668790235136/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827338.434547:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790235136/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827338.434552:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827338.434554:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827338.434556:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790235136:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827338.434559:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790235264:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827338.434569:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827338.434574:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827338/real 1787827338] req@ffff9e680a2d7c00 x1874668790235264/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827338.434581:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790235264/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827338.434591:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827338.434593:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827338.434595:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790235264:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827338.434598:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790235392:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827338.434609:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827338.434614:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827338/real 1787827338] req@ffff9e680a2d70c0 x1874668790235392/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827338.434620:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790235392/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827338.434625:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827338.434627:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827338.434629:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790235392:192.168.203.109@tcp:8:kworker.0 00010000:00010000:1.0:1787827339.137936:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827343.233906:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827343.554049:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790235520:192.168.203.109@tcp:250:kworker.0 00000800:00000100:0.0:1787827343.554145:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827343.554234:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827343/real 1787827343] req@ffff9e680a2d61c0 x1874668790235520/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827343.554300:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790235520/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827343.554337:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827343.554349:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827343.554363:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790235520:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827343.554388:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790235648:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827343.554446:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827343.554472:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827343/real 1787827343] req@ffff9e680a2d5e00 x1874668790235648/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827343.554498:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790235648/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827343.554540:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827343.554553:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827343.554564:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790235648:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827343.554618:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790235776:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827343.554684:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827343.554771:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827343/real 1787827343] req@ffff9e680a2d4780 x1874668790235776/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827343.554795:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790235776/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827343.554807:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827343.554811:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827343.554816:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790235776:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827343.554828:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790235904:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827343.554859:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827343.554871:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827343/real 1787827343] req@ffff9e680a2d4f00 x1874668790235904/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827343.554886:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790235904/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827343.554898:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827343.554904:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827343.554910:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790235904:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827343.554921:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790236032:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827343.554956:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827343.554979:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827343/real 1787827343] req@ffff9e680a2d43c0 x1874668790236032/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827343.555004:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790236032/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827343.555017:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827343.555021:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827343.555025:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790236032:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827343.555033:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790236160:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827343.555060:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827343.555074:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827343/real 1787827343] req@ffff9e680a2d4000 x1874668790236160/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827343.555087:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790236160/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827343.555107:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827343.555111:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827343.555115:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790236160:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827343.555121:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790236288:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827343.555140:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827343.555150:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827343/real 1787827343] req@ffff9e680a2d70c0 x1874668790236288/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827343.555164:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790236288/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827343.555199:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827343.555204:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827343.555208:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790236288:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827343.555236:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790236416:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827343.555263:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827343.555277:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827343/real 1787827343] req@ffff9e680a2d7c00 x1874668790236416/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827343.555291:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790236416/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827343.555301:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827343.555305:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827343.555310:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790236416:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827343.555317:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790236544:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827343.555336:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827343.555346:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827343/real 1787827343] req@ffff9e680a2d7480 x1874668790236544/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827343.555360:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790236544/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827343.555369:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827343.555373:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827343.555376:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790236544:192.168.203.109@tcp:8:kworker.0 00010000:00010000:1.0:1787827347.329862:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827348.673928:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790236672:192.168.203.109@tcp:250:kworker.0 00000800:00000100:0.0:1787827348.674001:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827348.674042:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827348/real 1787827348] req@ffff9e680a2d5a40 x1874668790236672/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827348.674061:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790236672/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827348.674079:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827348.674086:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827348.674093:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790236672:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827348.674119:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790236800:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827348.674159:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827348.674179:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827348/real 1787827348] req@ffff9e680a2d6d00 x1874668790236800/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827348.674193:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790236800/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827348.674209:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827348.674212:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827348.674215:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790236800:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827348.674221:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790236928:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827348.674235:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827348.674241:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827348/real 1787827348] req@ffff9e680a2d6580 x1874668790236928/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827348.674249:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790236928/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827348.674254:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827348.674256:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827348.674276:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790236928:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827348.674282:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790237056:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827348.674301:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827348.674310:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827348/real 1787827348] req@ffff9e680a2d52c0 x1874668790237056/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827348.674321:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790237056/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827348.674345:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827348.674349:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827348.674352:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790237056:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827348.674358:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790237184:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827348.674378:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827348.674390:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827348/real 1787827348] req@ffff9e680a2d4b40 x1874668790237184/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827348.674403:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790237184/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827348.674411:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827348.674414:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827348.674417:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790237184:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827348.674424:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790237312:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827348.674441:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827348.674450:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827348/real 1787827348] req@ffff9e680a2d7480 x1874668790237312/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827348.674461:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790237312/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827348.674470:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827348.674473:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827348.674477:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790237312:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827348.674482:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790237440:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827348.674497:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827348.674507:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827348/real 1787827348] req@ffff9e680a2d7c00 x1874668790237440/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827348.674517:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790237440/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827348.674525:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827348.674528:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827348.674531:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790237440:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827348.674554:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790237568:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827348.674571:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827348.674581:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827348/real 1787827348] req@ffff9e680a2d70c0 x1874668790237568/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827348.674592:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790237568/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827348.674600:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827348.674604:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827348.674607:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790237568:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827348.674613:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790237696:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827348.674628:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827348.674639:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827348/real 1787827348] req@ffff9e680a2d4000 x1874668790237696/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827348.674660:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790237696/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827348.674668:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827348.674671:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827348.674675:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790237696:192.168.203.109@tcp:8:kworker.0 00010000:00010000:1.0:1787827351.425883:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827353.794019:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790237824:192.168.203.109@tcp:250:kworker.0 00000800:00000100:0.0:1787827353.794131:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827353.794195:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827353/real 1787827353] req@ffff9e680a2d43c0 x1874668790237824/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827353.794260:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790237824/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827353.794299:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827353.794308:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827353.794318:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790237824:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827353.794330:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790237952:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827353.794368:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827353.794387:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827353/real 1787827353] req@ffff9e680a2d4f00 x1874668790237952/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827353.794396:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790237952/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827353.794408:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827353.794411:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827353.794413:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790237952:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827353.794433:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790238080:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827353.794445:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827353.794452:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827353/real 1787827353] req@ffff9e680a2d4780 x1874668790238080/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827353.794459:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790238080/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827353.794464:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827353.794466:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827353.794468:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790238080:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827353.794472:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790238208:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827353.794482:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827353.794489:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827353/real 1787827353] req@ffff9e680a2d5e00 x1874668790238208/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827353.794501:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790238208/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827353.794509:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827353.794513:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827353.794517:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790238208:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827353.794523:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790238336:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827353.794543:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827353.794554:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827353/real 1787827353] req@ffff9e680a2d61c0 x1874668790238336/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827353.794563:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790238336/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827353.794569:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827353.794571:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827353.794573:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790238336:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827353.794577:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790238464:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827353.794590:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827353.794597:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827353/real 1787827353] req@ffff9e680a2d4000 x1874668790238464/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827353.794605:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790238464/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827353.794611:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827353.794613:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827353.794615:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790238464:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827353.794618:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790238592:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827353.794638:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827353.794645:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827353/real 1787827353] req@ffff9e680a2d70c0 x1874668790238592/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827353.794653:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790238592/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827353.794658:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827353.794660:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827353.794662:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790238592:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827353.794666:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790238720:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827353.794676:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827353.794681:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827353/real 1787827353] req@ffff9e680a2d7c00 x1874668790238720/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827353.794688:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790238720/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827353.794720:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827353.794723:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827353.794725:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790238720:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827353.794731:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790238848:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827353.794751:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827353.794762:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827353/real 1787827353] req@ffff9e680a2d7480 x1874668790238848/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827353.794775:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790238848/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827353.794783:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827353.794786:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827353.794788:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790238848:192.168.203.109@tcp:8:kworker.0 00010000:00010000:1.0:1787827355.521864:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827358.913935:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790238976:192.168.203.109@tcp:250:kworker.0 00000800:00000100:0.0:1787827358.914024:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827358.914084:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827358/real 1787827358] req@ffff9e680a2d4b40 x1874668790238976/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827358.914116:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790238976/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827358.914137:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827358.914151:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827358.914163:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790238976:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827358.914196:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790239104:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827358.914230:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827358.914251:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827358/real 1787827358] req@ffff9e680a2d52c0 x1874668790239104/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827358.914299:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790239104/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827358.914332:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827358.914342:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827358.914353:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790239104:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827358.914385:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790239232:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827358.914421:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827358.914442:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827358/real 1787827358] req@ffff9e680a2d6580 x1874668790239232/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827358.914468:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790239232/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827358.914484:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827358.914490:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827358.914498:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790239232:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827358.914521:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790239360:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827358.914570:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827358.914590:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827358/real 1787827358] req@ffff9e680a2d6d00 x1874668790239360/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827358.914600:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790239360/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827358.914608:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827358.914610:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827358.914613:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790239360:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827358.914635:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790239488:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827358.914655:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827358.914665:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827358/real 1787827358] req@ffff9e680a2d5a40 x1874668790239488/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827358.914676:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790239488/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827358.914684:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827358.914687:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827358.914690:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790239488:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827358.914774:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790239616:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827358.914802:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827358.914816:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827358/real 1787827358] req@ffff9e680a2d4b40 x1874668790239616/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827358.914829:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790239616/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827358.914837:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827358.914839:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827358.914842:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790239616:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827358.914846:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790239744:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827358.914863:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827358.914871:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827358/real 1787827358] req@ffff9e680a2d7480 x1874668790239744/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827358.914879:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790239744/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827358.914891:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827358.914893:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827358.914896:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790239744:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827358.914900:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790239872:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827358.914911:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827358.914917:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827358/real 1787827358] req@ffff9e680a2d7c00 x1874668790239872/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827358.914925:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790239872/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827358.914931:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827358.914934:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827358.914936:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790239872:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827358.914957:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790240000:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827358.914973:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827358.914982:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827358/real 1787827358] req@ffff9e680a2d70c0 x1874668790240000/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827358.914993:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790240000/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827358.915000:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827358.915002:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827358.915006:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790240000:192.168.203.109@tcp:8:kworker.0 00010000:00010000:1.0:1787827359.617850:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827363.713870:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827364.033926:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790240128:192.168.203.109@tcp:250:kworker.0 00000800:00000100:0.0:1787827364.033990:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827364.034055:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827364/real 1787827364] req@ffff9e680a2d4000 x1874668790240128/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827364.034082:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790240128/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827364.034107:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827364.034116:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827364.034127:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790240128:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827364.034137:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790240256:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827364.034161:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827364.034171:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827364/real 1787827364] req@ffff9e680a2d61c0 x1874668790240256/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827364.034180:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790240256/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827364.034186:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827364.034188:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827364.034190:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790240256:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827364.034194:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790240384:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827364.034205:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827364.034212:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827364/real 1787827364] req@ffff9e680a2d5e00 x1874668790240384/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827364.034219:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790240384/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827364.034224:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827364.034226:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827364.034227:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790240384:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827364.034247:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790240512:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827364.034258:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827364.034264:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827364/real 1787827364] req@ffff9e680a2d4780 x1874668790240512/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827364.034271:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790240512/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827364.034276:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827364.034278:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827364.034280:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790240512:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827364.034283:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790240640:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827364.034292:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827364.034298:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827364/real 1787827364] req@ffff9e680a2d4f00 x1874668790240640/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827364.034305:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790240640/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827364.034310:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827364.034312:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827364.034314:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790240640:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827364.034317:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790240768:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827364.034326:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827364.034332:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827364/real 1787827364] req@ffff9e680a2d43c0 x1874668790240768/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827364.034338:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790240768/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827364.034344:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827364.034346:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827364.034348:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790240768:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827364.034351:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790240896:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827364.034360:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827364.034366:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827364/real 1787827364] req@ffff9e680a2d70c0 x1874668790240896/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827364.034372:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790240896/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827364.034377:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827364.034379:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827364.034381:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790240896:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827364.034385:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790241024:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827364.034394:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827364.034400:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827364/real 1787827364] req@ffff9e680a2d7c00 x1874668790241024/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827364.034406:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790241024/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827364.034412:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827364.034413:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827364.034415:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790241024:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827364.034418:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790241152:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827364.034427:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827364.034432:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827364/real 1787827364] req@ffff9e680a2d7480 x1874668790241152/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827364.034439:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790241152/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827364.034444:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827364.034445:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827364.034447:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790241152:192.168.203.109@tcp:8:kworker.0 00010000:00010000:1.0:1787827367.809897:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827369.153907:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790241280:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827369.154011:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790241408:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827369.154063:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790241536:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827369.154104:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790241664:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827369.154133:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790241792:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827369.154170:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790241920:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827369.154192:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790242048:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827369.154207:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790242176:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827369.154238:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790242304:192.168.203.109@tcp:8:kworker.0 00010000:00010000:1.0:1787827371.905936:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827372.226250:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827369/real 1787827372] req@ffff9e680a2d7c00 x1874668790242048/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827372.226308:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790242048/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827372.226552:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827372.226564:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827372.226582:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790242048:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827372.226600:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827369/real 1787827372] req@ffff9e680a2d70c0 x1874668790242176/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827372.226631:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790242176/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827372.226648:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827372.226657:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827372.226663:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790242176:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827372.226677:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827369/real 1787827372] req@ffff9e680a2d43c0 x1874668790242304/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827372.226772:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790242304/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827372.226790:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827372.226797:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827372.226804:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790242304:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827372.226817:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827369/real 1787827372] req@ffff9e680a2d6d00 x1874668790241536/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827372.226841:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790241536/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827372.226857:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827372.226864:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827372.226871:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790241536:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827372.226885:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827369/real 1787827372] req@ffff9e680a2d6580 x1874668790241664/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827372.226925:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790241664/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827372.226942:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827372.226951:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827372.226957:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790241664:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827372.226992:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827369/real 1787827372] req@ffff9e680a2d52c0 x1874668790241792/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827372.227018:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790241792/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827372.227035:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827372.227042:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827372.227048:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790241792:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827372.227061:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827369/real 1787827372] req@ffff9e680a2d7480 x1874668790241920/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827372.227085:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790241920/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827372.227104:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827372.227111:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827372.227118:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790241920:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827372.227131:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827369/real 1787827372] req@ffff9e680a2d4b40 x1874668790241280/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827372.227157:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790241280/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827372.227172:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827372.227179:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827372.227185:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790241280:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827372.227198:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827369/real 1787827372] req@ffff9e680a2d5a40 x1874668790241408/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827372.227222:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790241408/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827372.227239:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827372.227245:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827372.227252:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790241408:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827374.273935:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790242432:192.168.203.109@tcp:250:kworker.0 00000800:00000100:0.0:1787827374.274027:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827374.274087:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827374/real 1787827374] req@ffff9e680a2d4f00 x1874668790242432/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827374.274113:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790242432/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827374.274139:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827374.274152:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827374.274162:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790242432:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827374.274176:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790242560:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827374.274212:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827374.274229:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827374/real 1787827374] req@ffff9e680a2d4780 x1874668790242560/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827374.274273:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790242560/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827374.274291:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827374.274299:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827374.274303:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790242560:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827374.274331:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790242688:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827374.274356:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827374.274367:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827374/real 1787827374] req@ffff9e680a2d5e00 x1874668790242688/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827374.274379:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790242688/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827374.274387:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827374.274391:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827374.274395:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790242688:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827374.274401:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790242816:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827374.274421:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827374.274431:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827374/real 1787827374] req@ffff9e680a2d61c0 x1874668790242816/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827374.274443:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790242816/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827374.274457:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827374.274461:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827374.274465:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790242816:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827374.274470:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790242944:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827374.274487:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827374.274498:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827374/real 1787827374] req@ffff9e680a2d4000 x1874668790242944/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827374.274509:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790242944/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827374.274517:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827374.274521:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827374.274524:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790242944:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827374.274530:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790243072:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827374.274548:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827374.274558:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827374/real 1787827374] req@ffff9e680a2d5a40 x1874668790243072/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827374.274570:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790243072/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827374.274580:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827374.274584:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827374.274588:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790243072:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827374.274594:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790243200:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827374.274614:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827374.274625:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827374/real 1787827374] req@ffff9e680a2d4b40 x1874668790243200/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827374.274637:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790243200/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827374.274645:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827374.274649:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827374.274652:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790243200:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827374.274658:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790243328:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827374.274677:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827374.274688:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827374/real 1787827374] req@ffff9e680a2d7480 x1874668790243328/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827374.274738:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790243328/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827374.274746:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827374.274750:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827374.274753:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790243328:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827374.274778:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790243456:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827374.274800:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827374.274813:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827374/real 1787827374] req@ffff9e680a2d52c0 x1874668790243456/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827374.274826:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790243456/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827374.274833:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827374.274837:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827374.274840:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790243456:192.168.203.109@tcp:8:kworker.0 00010000:00010000:1.0:1787827376.001869:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827379.393797:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827379.393807:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827379.393812:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827379.393837:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4016 00000100:00080000:1.0:1787827379.393845:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827379.393851:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827379.393883:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790243584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827379.393905:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827379.393908:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827379.393909:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827379.393912:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4016 00000100:00080000:1.0:1787827379.393914:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827379.393917:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827379.393936:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827379.393946:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790243712/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827379.393954:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827379.393957:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827379.393958:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827379.393961:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4016 00000100:00080000:1.0:1787827379.393963:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827379.393967:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827379.393971:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790243584:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787827379.393972:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827379.393982:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790243840/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827379.393989:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827379.393991:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827379.393993:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827379.393995:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4016 00000100:00080000:1.0:1787827379.393997:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827379.393999:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827379.394002:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827379.394011:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790243968/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827379.394017:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827379.394021:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827379.394023:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827379.394026:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4016 00000100:00080000:1.0:1787827379.394030:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827379.394034:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827379.394039:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787827379.394040:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827379.394058:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790244096/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827379.394069:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827379.394073:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827379.394075:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827379.394080:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827379/real 1787827379] req@ffff9e6831aa8000 x1874668790243584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827379.394081:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4016 00000100:00080000:1.0:1787827379.394090:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827379.394096:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827379.394101:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827379.394103:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790243584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827379.394112:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790244224/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827379.394113:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827379.394122:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827379.394130:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790243584:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827379.394130:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827379.394137:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827379.394139:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827379.394141:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790243712:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827379.394145:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4016 00000100:00080000:1.0:1787827379.394149:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827379.394152:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827379.394157:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787827379.394166:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827379.394170:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790244352/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827379.394178:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827379.394179:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827379/real 1787827379] req@ffff9e6831aab480 x1874668790243712/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827379.394182:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827379.394185:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827379.394193:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4016 00000100:00080000:1.0:1787827379.394199:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827379.394204:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827379.394205:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790243712/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787827379.394208:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827379.394219:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790244480/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827379.394231:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827379.394235:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827379.394238:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787827379.394241:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787827379.394244:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790243712:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827379.394245:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827379.394254:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4016 00000100:00080000:1.0:1787827379.394256:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827379.394259:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827379.394263:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827379.394269:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790244608/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827379.394273:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790243840:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827379.394297:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827379.394306:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827379/real 1787827379] req@ffff9e6831aa9a40 x1874668790243840/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827379.394313:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790243840/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827379.394319:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827379.394321:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827379.394323:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790243840:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827379.394357:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790243968:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827379.394378:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827379.394389:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827379/real 1787827379] req@ffff9e6831aa8b40 x1874668790243968/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827379.394417:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790243968/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827379.394425:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827379.394428:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827379.394431:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790243968:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827379.394437:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790244096:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827379.394454:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827379.394480:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827379/real 1787827379] req@ffff9e6831aa8f00 x1874668790244096/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827379.394491:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790244096/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827379.394499:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827379.394502:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827379.394505:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790244096:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827379.394510:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790244224:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827379.394525:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827379.394534:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827379/real 1787827379] req@ffff9e6831aaa1c0 x1874668790244224/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827379.394545:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790244224/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827379.394554:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827379.394557:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827379.394560:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790244224:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827379.394565:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790244352:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827379.394597:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827379.394613:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827379/real 1787827379] req@ffff9e6831aa8780 x1874668790244352/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827379.394623:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790244352/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827379.394629:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827379.394631:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827379.394634:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790244352:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827379.394638:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790244480:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827379.394678:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827379.394688:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827379/real 1787827379] req@ffff9e6831aaa940 x1874668790244480/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827379.394699:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790244480/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827379.394708:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827379.394712:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827379.394715:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790244480:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827379.394756:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790244608:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827379.394841:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827379.394853:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827379/real 1787827379] req@ffff9e6831aaad00 x1874668790244608/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827379.394863:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790244608/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827379.394871:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827379.394873:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827379.394876:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790244608:192.168.203.109@tcp:8:kworker.0 00010000:00010000:1.0:1787827380.097913:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827384.193929:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827384.514129:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790244736:192.168.203.109@tcp:250:kworker.0 00000800:00000100:0.0:1787827384.514232:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827384.514289:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827384/real 1787827384] req@ffff9e680a2d6580 x1874668790244736/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827384.514315:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790244736/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827384.514327:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827384.514337:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827384.514343:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790244736:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827384.514382:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790244864:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827384.514406:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827384.514420:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827384/real 1787827384] req@ffff9e680a2d6d00 x1874668790244864/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827384.514439:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790244864/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827384.514453:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827384.514458:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827384.514463:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790244864:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827384.514477:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790244992:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827384.514494:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827384.514501:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827384/real 1787827384] req@ffff9e680a2d43c0 x1874668790244992/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827384.514509:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790244992/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827384.514514:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827384.514516:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827384.514518:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790244992:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827384.514522:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790245120:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827384.514533:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827384.514538:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827384/real 1787827384] req@ffff9e680a2d70c0 x1874668790245120/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827384.514545:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790245120/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827384.514551:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827384.514553:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827384.514554:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790245120:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827384.514558:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790245248:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827384.514567:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827384.514573:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827384/real 1787827384] req@ffff9e680a2d7c00 x1874668790245248/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827384.514580:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790245248/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827384.514586:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827384.514587:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827384.514589:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790245248:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827384.514592:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790245376:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827384.514603:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827384.514610:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827384/real 1787827384] req@ffff9e680a2d52c0 x1874668790245376/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827384.514623:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790245376/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827384.514633:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827384.514636:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827384.514640:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790245376:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827384.514646:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790245504:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827384.514681:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827384.514690:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827384/real 1787827384] req@ffff9e680a2d7480 x1874668790245504/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827384.514725:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790245504/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827384.514735:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827384.514738:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827384.514740:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790245504:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827384.514758:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790245632:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827384.514782:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827384.514789:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827384/real 1787827384] req@ffff9e680a2d4b40 x1874668790245632/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827384.514796:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790245632/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827384.514802:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827384.514804:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827384.514806:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790245632:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827384.514809:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790245760:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827384.514820:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827384.514825:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827384/real 1787827384] req@ffff9e680a2d5a40 x1874668790245760/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827384.514833:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790245760/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827384.514838:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827384.514840:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827384.514842:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790245760:192.168.203.109@tcp:8:kworker.0 00010000:00010000:1.0:1787827388.289944:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827389.633756:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827389.633765:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827389.633769:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827389.633776:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4026 00000100:00080000:0.0:1787827389.633785:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827389.633790:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827389.633820:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790245888/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827389.633844:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827389.633847:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827389.633848:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827389.633862:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4026 00000100:00080000:0.0:1787827389.633865:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827389.633868:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827389.633876:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827389.633887:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790246016/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827389.633904:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827389.633906:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827389.633908:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827389.633911:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4026 00000100:00080000:0.0:1787827389.633914:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827389.633917:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827389.633920:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827389.633929:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790246144/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827389.633936:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827389.633939:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827389.633940:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827389.633943:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4026 00000100:00080000:0.0:1787827389.633945:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827389.633947:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827389.633951:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827389.633960:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790246272/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827389.633968:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827389.633971:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827389.633972:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827389.633974:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4026 00000100:00080000:0.0:1787827389.633976:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827389.633979:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827389.633982:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827389.633992:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790246400/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827389.634006:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827389.634009:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827389.634010:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827389.634017:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4026 00000100:00080000:0.0:1787827389.634020:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827389.634023:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827389.634026:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827389.634035:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790246528/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827389.634045:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827389.634050:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827389.634054:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827389.634066:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4026 00000100:00080000:0.0:1787827389.634070:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827389.634075:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827389.634081:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827389.634099:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790246656/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827389.634118:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827389.634122:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827389.634124:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827389.634132:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4026 00000100:00080000:0.0:1787827389.634138:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827389.634142:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827389.634146:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827389.634157:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790246784/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827389.634184:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827389.634187:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827389.634190:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827389.634195:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4026 00000100:00080000:0.0:1787827389.634206:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827389.634213:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827389.634218:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827389.634230:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790246912/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827392.386097:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827396.481893:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827399.873882:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827399.873902:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827399.873910:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827399.873928:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4036 00000100:00080000:1.0:1787827399.873949:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827399.873963:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827399.874030:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790248192/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827399.874077:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827399.874084:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827399.874087:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827399.874094:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4036 00000100:00080000:1.0:1787827399.874100:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827399.874107:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827399.874124:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827399.874145:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790248320/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827399.874164:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827399.874171:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827399.874175:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827399.874182:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4036 00000100:00080000:1.0:1787827399.874189:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827399.874198:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827399.874220:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827399.874253:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790248448/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827399.874279:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827399.874286:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827399.874289:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827399.874295:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4036 00000100:00080000:1.0:1787827399.874301:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827399.874309:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827399.874320:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827399.874363:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790248576/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827399.874387:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827399.874393:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827399.874397:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827399.874404:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4036 00000100:00080000:1.0:1787827399.874413:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827399.874421:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827399.874436:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827399.874463:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790248704/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827399.874512:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827399.874524:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827399.874531:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827399.874549:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4036 00000100:00080000:1.0:1787827399.874558:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827399.874574:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827399.874587:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827399.874616:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790248832/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827399.874637:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827399.874644:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827399.874655:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827399.874670:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4036 00000100:00080000:1.0:1787827399.874680:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827399.874770:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827399.874833:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827399.874884:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790248960/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827399.874937:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827399.874945:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827399.874950:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827399.874971:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4036 00000100:00080000:1.0:1787827399.874983:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827399.874993:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827399.875005:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827399.875034:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790249088/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827399.875057:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827399.875070:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827399.875074:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827399.875090:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4036 00000100:00080000:1.0:1787827399.875112:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827399.875129:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827399.875157:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827399.875200:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790249216/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827400.577883:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827404.674046:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827404.994150:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790249344:192.168.203.109@tcp:250:kworker.0 00000800:00000100:0.0:1787827404.994261:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827404.994320:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827404/real 1787827404] req@ffff9e680a2d52c0 x1874668790249344/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827404.994358:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790249344/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827404.994373:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827404.994378:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827404.994384:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790249344:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827404.994423:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790249472:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827404.994481:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827404.994525:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827404/real 1787827404] req@ffff9e680a2d7c00 x1874668790249472/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827404.994567:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790249472/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827404.994603:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827404.994611:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827404.994618:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790249472:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827404.994634:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790249600:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827404.994687:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827404.994789:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827404/real 1787827404] req@ffff9e680a2d70c0 x1874668790249600/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827404.994810:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790249600/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827404.994824:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827404.994829:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827404.994836:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790249600:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827404.994993:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790249728:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827404.995087:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827404.995151:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827404/real 1787827404] req@ffff9e680a2d43c0 x1874668790249728/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827404.995214:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790249728/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827404.995239:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827404.995252:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827404.995269:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790249728:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827404.995308:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790249856:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827404.995347:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827404.995370:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827404/real 1787827404] req@ffff9e680a2d6d00 x1874668790249856/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827404.995407:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790249856/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827404.995446:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827404.995460:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827404.995475:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790249856:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827404.995509:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790249984:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827404.995576:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827404.995613:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827404/real 1787827404] req@ffff9e680a2d6580 x1874668790249984/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827404.995639:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790249984/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827404.995661:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827404.995665:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827404.995669:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790249984:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827404.995691:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790250112:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827404.995797:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827404.995816:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827404/real 1787827404] req@ffff9e680a2d43c0 x1874668790250112/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827404.995832:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790250112/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827404.995845:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827404.995850:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827404.995857:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790250112:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827404.995870:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790250240:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827404.995895:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827404.995906:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827404/real 1787827404] req@ffff9e680a2d70c0 x1874668790250240/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827404.995920:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790250240/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827404.995929:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827404.995933:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827404.995937:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790250240:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827404.995944:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790250368:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827404.995963:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827404.995974:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827404/real 1787827404] req@ffff9e680a2d7c00 x1874668790250368/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827404.995987:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790250368/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827404.995996:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827404.996000:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827404.996006:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790250368:192.168.203.109@tcp:8:kworker.0 00010000:00010000:1.0:1787827408.769833:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827410.114179:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790250496:192.168.203.109@tcp:250:kworker.0 00000800:00000100:0.0:1787827410.114301:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827410.114366:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827410/real 1787827410] req@ffff9e680a2d52c0 x1874668790250496/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827410.114456:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790250496/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827410.114500:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827410.114517:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827410.114525:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790250496:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827410.114577:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790250624:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827410.114642:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827410.114685:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827410/real 1787827410] req@ffff9e680a2d7480 x1874668790250624/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827410.114796:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790250624/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827410.114812:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827410.114828:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827410.114840:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790250624:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827410.114856:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790250752:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827410.114904:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827410.114943:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827410/real 1787827410] req@ffff9e680a2d4b40 x1874668790250752/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827410.114992:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790250752/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827410.115029:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827410.115058:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827410.115064:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790250752:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827410.115091:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790250880:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827410.115120:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827410.115137:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827410/real 1787827410] req@ffff9e680a2d5a40 x1874668790250880/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827410.115160:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790250880/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827410.115175:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827410.115184:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827410.115188:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790250880:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827410.115199:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790251008:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827410.115220:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827410.115233:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827410/real 1787827410] req@ffff9e680a2d4000 x1874668790251008/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827410.115281:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790251008/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827410.115303:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827410.115313:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827410.115322:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790251008:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827410.115338:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790251136:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827410.115369:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827410.115408:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827410/real 1787827410] req@ffff9e680a2d4f00 x1874668790251136/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827410.115432:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790251136/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827410.115447:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827410.115452:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827410.115457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790251136:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827410.115482:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790251264:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827410.115506:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827410.115517:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827410/real 1787827410] req@ffff9e680a2d4780 x1874668790251264/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827410.115527:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790251264/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827410.115535:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827410.115538:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827410.115541:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790251264:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827410.115566:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790251392:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827410.115587:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827410.115598:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827410/real 1787827410] req@ffff9e680a2d5e00 x1874668790251392/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827410.115612:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790251392/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827410.115621:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827410.115625:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827410.115628:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790251392:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827410.115634:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790251520:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827410.115652:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827410.115669:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827410/real 1787827410] req@ffff9e680a2d61c0 x1874668790251520/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827410.115691:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790251520/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827410.115764:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827410.115774:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827410.115782:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790251520:192.168.203.109@tcp:8:kworker.0 00010000:00010000:1.0:1787827412.865927:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827415.234085:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790251648:192.168.203.109@tcp:250:kworker.0 00000800:00000100:0.0:1787827415.234215:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827415.234285:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827415/real 1787827415] req@ffff9e680a2d61c0 x1874668790251648/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827415.234342:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790251648/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827415.234388:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827415.234401:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827415.234415:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790251648:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827415.234441:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790251776:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827415.234484:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827415.234529:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827415/real 1787827415] req@ffff9e680a2d5e00 x1874668790251776/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827415.234572:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790251776/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827415.234628:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827415.234643:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827415.234653:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790251776:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827415.234765:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790251904:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827415.234822:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827415.234849:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827415/real 1787827415] req@ffff9e680a2d4780 x1874668790251904/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827415.234870:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790251904/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827415.234881:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827415.234886:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827415.234890:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790251904:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827415.234898:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790252032:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827415.234918:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827415.234941:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827415/real 1787827415] req@ffff9e680a2d4f00 x1874668790252032/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827415.234954:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790252032/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827415.234964:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827415.234967:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827415.234972:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790252032:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827415.234978:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790252160:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827415.234996:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827415.235007:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827415/real 1787827415] req@ffff9e680a2d4000 x1874668790252160/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827415.235020:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790252160/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827415.235039:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827415.235042:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827415.235046:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790252160:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827415.235052:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790252288:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827415.235070:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827415.235082:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827415/real 1787827415] req@ffff9e680a2d5a40 x1874668790252288/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827415.235095:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790252288/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827415.235107:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827415.235110:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827415.235114:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790252288:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827415.235120:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790252416:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827415.235139:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827415.235158:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827415/real 1787827415] req@ffff9e680a2d4b40 x1874668790252416/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827415.235172:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790252416/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827415.235181:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827415.235185:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827415.235188:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790252416:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827415.235194:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790252544:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827415.235212:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827415.235222:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827415/real 1787827415] req@ffff9e680a2d7480 x1874668790252544/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827415.235235:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790252544/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827415.235244:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827415.235248:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827415.235251:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790252544:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827415.235274:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790252672:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827415.235293:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827415.235305:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827415/real 1787827415] req@ffff9e680a2d52c0 x1874668790252672/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827415.235318:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790252672/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827415.235329:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827415.235332:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827415.235336:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790252672:192.168.203.109@tcp:8:kworker.0 00010000:00010000:1.0:1787827416.961954:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827420.353843:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827420.353855:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827420.353860:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827420.353871:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4057 00000100:00080000:0.0:1787827420.353883:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827420.353909:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827420.353958:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790252800/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827420.354001:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827420.354006:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827420.354008:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827420.354013:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4057 00000100:00080000:0.0:1787827420.354034:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827420.354038:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827420.354048:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827420.354061:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790252928/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827420.354076:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827420.354082:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827420.354084:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827420.354089:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4057 00000100:00080000:0.0:1787827420.354091:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827420.354094:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827420.354099:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827420.354111:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790253056/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827420.354121:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827420.354125:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827420.354126:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827420.354129:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4057 00000100:00080000:0.0:1787827420.354134:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827420.354137:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827420.354142:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827420.354158:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790253184/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827420.354168:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827420.354171:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827420.354172:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827420.354176:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4057 00000100:00080000:0.0:1787827420.354179:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827420.354183:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827420.354188:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827420.354205:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790253312/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827420.354218:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827420.354221:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827420.354222:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827420.354226:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4057 00000100:00080000:0.0:1787827420.354232:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827420.354240:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827420.354247:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827420.354261:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790253440/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827420.354277:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827420.354280:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827420.354282:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827420.354289:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4057 00000100:00080000:0.0:1787827420.354292:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827420.354300:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827420.354308:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827420.354328:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790253568/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827420.354354:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827420.354359:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827420.354363:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827420.354375:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4057 00000100:00080000:0.0:1787827420.354386:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827420.354392:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827420.354400:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827420.354441:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790253696/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827420.354462:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827420.354467:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827420.354469:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827420.354515:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4057 00000100:00080000:0.0:1787827420.354522:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827420.354535:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827420.354565:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827420.354587:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790253824/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827421.057963:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827425.153903:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827429.249928:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827430.593808:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827430.593817:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827430.593820:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827430.593828:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4067 00000100:00080000:1.0:1787827430.593836:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827430.593842:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827430.593875:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790255104/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827430.593895:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827430.593898:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827430.593899:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827430.593902:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4067 00000100:00080000:1.0:1787827430.593904:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827430.593907:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827430.593915:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827430.593924:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790255232/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827430.593933:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827430.593936:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827430.593940:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827430.593947:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4067 00000100:00080000:1.0:1787827430.593949:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827430.593951:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827430.593954:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827430.593963:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790255360/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827430.593970:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827430.593973:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827430.593974:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827430.593977:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4067 00000100:00080000:1.0:1787827430.593980:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827430.593982:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827430.593985:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827430.593994:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790255488/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827430.594007:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827430.594010:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827430.594011:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827430.594015:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4067 00000100:00080000:1.0:1787827430.594021:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827430.594023:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827430.594026:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827430.594036:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790255616/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827430.594046:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827430.594053:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827430.594056:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827430.594064:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4067 00000100:00080000:1.0:1787827430.594068:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827430.594073:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827430.594077:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827430.594094:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790255744/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827430.594114:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827430.594116:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827430.594118:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827430.594121:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4067 00000100:00080000:1.0:1787827430.594124:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827430.594129:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827430.594136:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827430.594147:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790255872/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827430.594155:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827430.594160:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827430.594163:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827430.594167:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4067 00000100:00080000:1.0:1787827430.594171:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827430.594177:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827430.594183:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827430.594195:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790256000/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827430.594210:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827430.594215:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827430.594219:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827430.594225:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4067 00000100:00080000:1.0:1787827430.594229:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827430.594234:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827430.594242:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827430.594250:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790256128/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827433.345890:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827435.713790:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827435.713803:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827435.713808:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827435.713819:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4072 00000100:00080000:1.0:1787827435.713847:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827435.713854:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827435.713891:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790256256/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827435.713918:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827435.713923:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827435.713925:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827435.713930:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4072 00000100:00080000:1.0:1787827435.713935:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827435.713940:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827435.713950:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827435.713978:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790256384/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827435.713991:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827435.713995:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827435.713997:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827435.714001:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4072 00000100:00080000:1.0:1787827435.714005:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827435.714010:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827435.714015:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827435.714025:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790256512/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827435.714036:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827435.714039:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827435.714040:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827435.714044:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4072 00000100:00080000:1.0:1787827435.714047:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827435.714051:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827435.714056:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827435.714067:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790256640/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827435.714077:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827435.714081:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827435.714083:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827435.714104:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4072 00000100:00080000:1.0:1787827435.714108:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827435.714112:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827435.714118:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827435.714130:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790256768/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827435.714140:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827435.714144:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827435.714146:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827435.714152:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4072 00000100:00080000:1.0:1787827435.714156:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827435.714160:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827435.714165:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827435.714178:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790256896/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827435.714188:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827435.714191:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827435.714193:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827435.714198:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4072 00000100:00080000:1.0:1787827435.714202:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827435.714207:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827435.714212:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827435.714224:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790257024/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827435.714236:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827435.714240:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827435.714243:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827435.714247:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4072 00000100:00080000:1.0:1787827435.714251:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827435.714256:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827435.714262:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827435.714274:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790257152/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827435.714286:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827435.714290:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827435.714292:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827435.714296:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4072 00000100:00080000:1.0:1787827435.714300:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827435.714304:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827435.714326:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827435.714338:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790257280/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827437.441911:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787827438.785935:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:0.0:1787827438.785991:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787827438.786029:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827438.786073:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:0.0:1787827438.786093:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827438.786104:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827438.786114:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827438.786125:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827438.786137:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827438.786147:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827438.786153:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00080000:0.0:1787827440.833824:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827440.833831:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827440.833834:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827440.833843:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4077 00000100:00080000:0.0:1787827440.833854:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827440.833877:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827440.833908:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790257408/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827440.833936:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827440.833939:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827440.833941:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827440.833945:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4077 00000100:00080000:0.0:1787827440.833950:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827440.833956:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827440.833965:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827440.833984:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790257536/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827440.833997:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827440.834001:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827440.834004:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827440.834012:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4077 00000100:00080000:0.0:1787827440.834021:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827440.834029:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827440.834034:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827440.834051:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790257664/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827440.834065:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827440.834069:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827440.834072:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827440.834084:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4077 00000100:00080000:0.0:1787827440.834093:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827440.834101:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827440.834108:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827440.834123:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790257792/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827440.834137:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827440.834162:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827440.834166:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827440.834177:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4077 00000100:00080000:0.0:1787827440.834185:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827440.834195:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827440.834204:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827440.834234:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790257920/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827440.834246:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827440.834253:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827440.834258:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827440.834263:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4077 00000100:00080000:0.0:1787827440.834269:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827440.834278:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827440.834288:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827440.834304:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790258048/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827440.834320:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827440.834333:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827440.834337:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827440.834343:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4077 00000100:00080000:0.0:1787827440.834347:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827440.834352:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827440.834356:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827440.834371:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790258176/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827440.834384:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827440.834388:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827440.834390:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827440.834394:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4077 00000100:00080000:0.0:1787827440.834402:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827440.834410:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827440.834417:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827440.834433:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790258304/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827440.834452:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827440.834457:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827440.834463:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827440.834473:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4077 00000100:00080000:0.0:1787827440.834481:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827440.834489:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827440.834513:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827440.834526:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790258432/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827441.537865:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827445.633855:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827445.953832:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827445.953846:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827445.953852:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827445.953865:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4082 00000100:00080000:1.0:1787827445.953880:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827445.953889:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827445.953961:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790258560/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827445.953993:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827445.953997:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827445.953999:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827445.954003:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4082 00000100:00080000:1.0:1787827445.954007:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827445.954010:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827445.954021:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827445.954036:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790258688/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827445.954047:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827445.954050:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827445.954052:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827445.954055:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4082 00000100:00080000:1.0:1787827445.954058:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827445.954065:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827445.954073:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827445.954083:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790258816/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827445.954092:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827445.954095:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827445.954097:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827445.954102:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4082 00000100:00080000:1.0:1787827445.954105:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827445.954108:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827445.954113:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827445.954124:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790258944/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827445.954131:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827445.954135:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827445.954137:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827445.954140:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4082 00000100:00080000:1.0:1787827445.954143:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827445.954146:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827445.954150:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827445.954175:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790259072/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827445.954184:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827445.954187:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827445.954189:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827445.954193:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4082 00000100:00080000:1.0:1787827445.954197:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827445.954200:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827445.954205:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827445.954220:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790259200/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827445.954251:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827445.954258:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827445.954262:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827445.954276:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4082 00000100:00080000:1.0:1787827445.954286:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827445.954294:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827445.954308:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827445.954331:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790259328/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827445.954347:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827445.954354:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827445.954361:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827445.954389:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4082 00000100:00080000:1.0:1787827445.954396:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827445.954403:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827445.954411:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827445.954431:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790259456/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827445.954457:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827445.954485:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827445.954491:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827445.954518:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4082 00000100:00080000:1.0:1787827445.954526:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827445.954538:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827445.954550:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827445.954573:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790259584/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827449.729916:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827451.073978:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790259712:192.168.203.109@tcp:250:kworker.0 00000800:00000100:0.0:1787827451.074043:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827451.074094:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827451/real 1787827451] req@ffff9e680a2d4000 x1874668790259712/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827451.074132:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790259712/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827451.074145:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827451.074151:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827451.074156:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790259712:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827451.074179:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790259840:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827451.074232:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827451.074247:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827451/real 1787827451] req@ffff9e680a2d4f00 x1874668790259840/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827451.074259:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790259840/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827451.074273:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827451.074278:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827451.074283:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790259840:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827451.074293:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790259968:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827451.074335:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827451.074345:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827451/real 1787827451] req@ffff9e680a2d4780 x1874668790259968/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827451.074359:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790259968/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827451.074398:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827451.074407:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827451.074416:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790259968:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827451.074458:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790260096:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827451.074486:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827451.074498:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827451/real 1787827451] req@ffff9e680a2d5e00 x1874668790260096/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827451.074552:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790260096/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827451.074568:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827451.074576:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827451.074581:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790260096:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827451.074599:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790260224:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827451.074618:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827451.074634:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827451/real 1787827451] req@ffff9e680a2d61c0 x1874668790260224/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827451.074648:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790260224/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827451.074673:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827451.074676:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827451.074679:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790260224:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827451.074686:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790260352:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827451.074706:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827451.074718:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827451/real 1787827451] req@ffff9e680a2d4000 x1874668790260352/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827451.074751:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790260352/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827451.074762:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827451.074764:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827451.074767:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790260352:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827451.074771:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790260480:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827451.074786:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827451.074793:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827451/real 1787827451] req@ffff9e680a2d5a40 x1874668790260480/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827451.074800:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790260480/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827451.074805:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827451.074807:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827451.074809:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790260480:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827451.074813:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790260608:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827451.074831:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827451.074838:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827451/real 1787827451] req@ffff9e680a2d4b40 x1874668790260608/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827451.074845:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790260608/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827451.074851:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827451.074853:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827451.074855:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790260608:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827451.074866:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790260736:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827451.074886:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827451.074897:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827451/real 1787827451] req@ffff9e680a2d7480 x1874668790260736/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827451.074910:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790260736/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827451.074919:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827451.074922:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827451.074926:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790260736:192.168.203.109@tcp:8:kworker.0 00010000:00010000:1.0:1787827453.825927:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827456.193774:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827456.193782:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827456.193786:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827456.193795:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4092 00000100:00080000:1.0:1787827456.193805:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827456.193812:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827456.193843:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790260864/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827456.193864:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827456.193866:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827456.193868:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827456.193870:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4092 00000100:00080000:1.0:1787827456.193872:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827456.193875:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827456.193882:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827456.193892:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790260992/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827456.193905:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827456.193908:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790260864:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827456.193909:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827456.193912:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827456.193919:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4092 00000100:00080000:1.0:1787827456.193922:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827456.193924:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827456.193927:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827456.193942:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790261120/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827456.193949:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827456.193952:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827456.193953:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827456.193955:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4092 00000100:00080000:1.0:1787827456.193960:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827456.193963:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827456.193966:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787827456.193970:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827456.193976:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790261248/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827456.193984:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827456.193986:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827456.193987:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827456.193990:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4092 00000100:00080000:1.0:1787827456.193993:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827456.193996:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827456.193999:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827456.194005:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827456/real 1787827456] req@ffff9e6831aaad00 x1874668790260864/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827456.194008:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790261376/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827456.194029:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827456.194032:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827456.194033:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827456.194039:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4092 00000100:00080000:1.0:1787827456.194041:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827456.194043:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827456.194046:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790260864/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787827456.194046:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827456.194056:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790261504/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827456.194062:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827456.194063:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827456.194068:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827456.194070:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787827456.194070:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827456.194078:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790260864:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827456.194084:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4092 00000100:00080000:1.0:1787827456.194088:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827456.194091:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827456.194094:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827456.194100:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790260992:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827456.194103:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790261632/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827456.194116:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827456.194119:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827456.194120:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787827456.194124:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827456.194127:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4092 00000100:00080000:1.0:1787827456.194132:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827456.194135:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827456.194138:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827456.194140:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827456/real 1787827456] req@ffff9e6831aa8780 x1874668790260992/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827456.194148:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790261760/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827456.194155:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790260992/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827456.194157:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827456.194164:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827456.194166:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827456.194175:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827456.194178:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4092 00000100:00080000:0.0:1787827456.194179:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787827456.194183:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827456.194185:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790260992:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827456.194190:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827456.194195:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790261120:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787827456.194195:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827456.194207:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790261888/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827456.194216:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827456.194226:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827456/real 1787827456] req@ffff9e6831aaa1c0 x1874668790261120/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827456.194234:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790261120/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827456.194239:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827456.194241:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827456.194244:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790261120:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827456.194247:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790261248:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827456.194258:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827456.194263:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827456/real 1787827456] req@ffff9e6831aa8f00 x1874668790261248/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827456.194270:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790261248/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827456.194276:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827456.194278:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827456.194280:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790261248:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827456.194283:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790261376:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827456.194292:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827456.194297:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827456/real 1787827456] req@ffff9e6831aaa940 x1874668790261376/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827456.194303:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790261376/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827456.194308:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827456.194310:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827456.194312:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790261376:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827456.194316:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790261504:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827456.194325:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827456.194330:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827456/real 1787827456] req@ffff9e6831aaa580 x1874668790261504/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827456.194336:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790261504/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827456.194342:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827456.194345:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827456.194347:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790261504:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827456.194358:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790261632:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827456.194368:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827456.194374:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827456/real 1787827456] req@ffff9e6831aa9e00 x1874668790261632/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827456.194381:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790261632/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827456.194385:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827456.194401:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827456.194404:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790261632:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827456.194408:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790261760:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827456.194422:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827456.194430:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827456/real 1787827456] req@ffff9e6831aabc00 x1874668790261760/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827456.194437:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790261760/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827456.194442:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827456.194444:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827456.194446:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790261760:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827456.194449:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790261888:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827456.194459:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827456.194465:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827456/real 1787827456] req@ffff9e6831aa9680 x1874668790261888/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827456.194472:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790261888/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827456.194477:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827456.194479:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827456.194481:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790261888:192.168.203.109@tcp:8:kworker.0 00010000:00010000:1.0:1787827457.921947:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827461.313793:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827461.313805:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827461.313810:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827461.313820:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4098 00000100:00080000:1.0:1787827461.313832:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827461.313840:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827461.313884:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790262016/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827461.313912:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827461.313915:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827461.313916:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827461.313919:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4098 00000100:00080000:1.0:1787827461.313922:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827461.313925:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827461.313933:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827461.313944:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790262144/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827461.313952:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827461.313955:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827461.313956:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827461.313958:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4098 00000100:00080000:1.0:1787827461.313960:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827461.313964:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790262016:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827461.313966:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827461.313975:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827461.313982:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790262272/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827461.313989:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827461.313993:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827461.313995:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827461.313998:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4098 00000100:00080000:1.0:1787827461.314013:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827461.314016:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827461.314019:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787827461.314023:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827461.314031:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790262400/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827461.314039:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827461.314041:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827461.314042:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827461.314044:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4098 00000100:00080000:1.0:1787827461.314047:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827461.314049:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827461.314053:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827461.314060:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827461/real 1787827461] req@ffff9e6831aa8b40 x1874668790262016/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827461.314063:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790262528/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827461.314079:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827461.314087:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827461.314090:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827461.314094:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790262016/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827461.314099:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4098 00000100:00080000:1.0:1787827461.314108:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827461.314110:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827461.314116:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787827461.314120:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827461.314122:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790262016:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787827461.314132:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827461.314133:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790262144:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827461.314155:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790262656/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827461.314161:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827461.314169:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827461.314174:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787827461.314175:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827461/real 1787827461] req@ffff9e6831aa9a40 x1874668790262144/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827461.314176:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827461.314186:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4098 00000100:00080000:1.0:1787827461.314194:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827461.314198:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790262144/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827461.314201:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827461.314209:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787827461.314221:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787827461.314223:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790262784/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827461.314227:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827461.314234:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790262144:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827461.314241:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827461.314245:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827461.314246:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827461.314250:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4098 00000100:00080000:1.0:1787827461.314252:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827461.314254:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827461.314257:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827461.314261:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790262272:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827461.314267:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790262912/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827461.314276:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787827461.314280:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827461.314282:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827461.314285:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827461.314289:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4098 00000100:00080000:1.0:1787827461.314292:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827461.314293:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827461/real 1787827461] req@ffff9e6831aab480 x1874668790262272/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827461.314297:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827461.314305:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827461.314311:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790262272/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827461.314317:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790263040/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827461.314328:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827461.314336:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827461.314341:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790262272:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827461.314346:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790262400:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827461.314376:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827461.314385:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827461/real 1787827461] req@ffff9e6831aa8000 x1874668790262400/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827461.314393:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790262400/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827461.314398:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827461.314400:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827461.314403:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790262400:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827461.314407:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790262528:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827461.314418:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827461.314424:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827461/real 1787827461] req@ffff9e6831aa9680 x1874668790262528/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827461.314431:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790262528/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827461.314436:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827461.314438:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827461.314440:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790262528:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827461.314443:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790262656:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827461.314452:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827461.314458:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827461/real 1787827461] req@ffff9e6831aabc00 x1874668790262656/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827461.314466:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790262656/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827461.314472:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827461.314473:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827461.314475:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790262656:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827461.314479:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790262784:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827461.314488:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827461.314494:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827461/real 1787827461] req@ffff9e6831aa9e00 x1874668790262784/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827461.314501:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790262784/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827461.314506:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827461.314508:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827461.314510:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790262784:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827461.314524:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790262912:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827461.314543:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827461.314554:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827461/real 1787827461] req@ffff9e6831aaa580 x1874668790262912/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827461.314567:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790262912/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827461.314575:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827461.314578:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827461.314582:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790262912:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827461.314588:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790263040:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827461.314608:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827461.314620:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827461/real 1787827461] req@ffff9e6831aaa940 x1874668790263040/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827461.314628:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790263040/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827461.314634:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827461.314636:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827461.314638:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790263040:192.168.203.109@tcp:8:kworker.0 00010000:00010000:1.0:1787827462.017905:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827466.113913:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827466.433954:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790263168:192.168.203.109@tcp:250:kworker.0 00000800:00000100:0.0:1787827466.434030:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827466.434074:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827466/real 1787827466] req@ffff9e6804423480 x1874668790263168/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827466.434101:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790263168/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827466.434134:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827466.434147:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827466.434184:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790263168:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827466.434218:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790263296:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827466.434263:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827466.434282:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827466/real 1787827466] req@ffff9e68044230c0 x1874668790263296/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827466.434333:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790263296/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827466.434379:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827466.434391:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827466.434402:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790263296:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827466.434414:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790263424:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827466.434449:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827466.434460:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827466/real 1787827466] req@ffff9e6804422580 x1874668790263424/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827466.434470:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790263424/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827466.434477:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827466.434479:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827466.434482:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790263424:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827466.434486:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790263552:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827466.434499:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827466.434505:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827466/real 1787827466] req@ffff9e6804420b40 x1874668790263552/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827466.434514:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790263552/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827466.434520:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827466.434522:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827466.434524:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790263552:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827466.434528:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790263680:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827466.434539:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827466.434546:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827466/real 1787827466] req@ffff9e68044221c0 x1874668790263680/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827466.434555:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790263680/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827466.434561:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827466.434562:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827466.434565:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790263680:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827466.434568:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790263808:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827466.434590:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827466.434596:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827466/real 1787827466] req@ffff9e6804423840 x1874668790263808/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827466.434605:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790263808/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827466.434612:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827466.434614:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827466.434616:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790263808:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827466.434630:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790263936:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827466.434642:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827466.434649:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827466/real 1787827466] req@ffff9e68044212c0 x1874668790263936/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827466.434657:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790263936/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827466.434668:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827466.434670:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827466.434672:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790263936:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827466.434676:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790264064:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827466.434696:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827466.434702:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827466/real 1787827466] req@ffff9e6804421680 x1874668790264064/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827466.434733:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790264064/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827466.434740:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827466.434742:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827466.434744:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790264064:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827466.434749:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790264192:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827466.434764:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827466.434800:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827466/real 1787827466] req@ffff9e6804420780 x1874668790264192/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827466.434810:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790264192/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827466.434816:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827466.434819:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827466.434822:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790264192:192.168.203.109@tcp:8:kworker.0 00010000:00010000:1.0:1787827470.209950:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827471.553817:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827471.553828:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827471.553833:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827471.553843:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4108 00000100:00080000:0.0:1787827471.553863:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827471.553872:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827471.553914:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790264320/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827471.553939:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827471.553942:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827471.553944:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827471.553946:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4108 00000100:00080000:0.0:1787827471.553949:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827471.553952:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827471.553960:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827471.553969:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790264448/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827471.553977:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827471.553979:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827471.553981:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827471.553983:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4108 00000100:00080000:0.0:1787827471.553985:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827471.553988:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827471.553991:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827471.553999:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790264576/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827471.554018:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827471.554026:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827471.554028:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827471.554031:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4108 00000100:00080000:0.0:1787827471.554033:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827471.554036:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827471.554039:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827471.554048:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790264704/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827471.554058:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827471.554062:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827471.554064:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827471.554073:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4108 00000100:00080000:0.0:1787827471.554080:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827471.554085:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827471.554090:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827471.554105:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790264832/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827471.554126:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827471.554131:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827471.554133:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827471.554152:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4108 00000100:00080000:0.0:1787827471.554161:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827471.554171:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827471.554180:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827471.554194:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790264960/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827471.554207:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827471.554212:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827471.554215:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827471.554222:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4108 00000100:00080000:0.0:1787827471.554228:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827471.554233:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827471.554240:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827471.554251:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790265088/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827471.554266:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827471.554268:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827471.554269:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827471.554272:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4108 00000100:00080000:0.0:1787827471.554275:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827471.554277:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827471.554296:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827471.554308:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790265216/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827471.554322:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827471.554324:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827471.554326:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827471.554329:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4108 00000100:00080000:0.0:1787827471.554332:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827471.554339:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827471.554344:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827471.554355:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790265344/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827474.305881:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827476.673917:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790265472:192.168.203.109@tcp:250:kworker.0 00000800:00000100:0.0:1787827476.673994:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827476.674039:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827476/real 1787827476] req@ffff9e680a2d52c0 x1874668790265472/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827476.674074:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790265472/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827476.674085:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827476.674088:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827476.674092:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790265472:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827476.674124:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790265600:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827476.674160:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827476.674181:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827476/real 1787827476] req@ffff9e680a2d6d00 x1874668790265600/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827476.674201:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790265600/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827476.674225:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827476.674233:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827476.674239:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790265600:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827476.674268:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790265728:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827476.674289:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827476.674297:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827476/real 1787827476] req@ffff9e680a2d6580 x1874668790265728/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827476.674306:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790265728/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827476.674312:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827476.674314:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827476.674317:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790265728:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827476.674321:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790265856:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827476.674332:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827476.674339:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827476/real 1787827476] req@ffff9e680a2d43c0 x1874668790265856/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827476.674352:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790265856/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827476.674358:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827476.674360:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827476.674362:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790265856:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827476.674365:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790265984:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827476.674377:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827476.674386:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827476/real 1787827476] req@ffff9e680a2d70c0 x1874668790265984/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827476.674403:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790265984/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827476.674416:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827476.674419:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827476.674423:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790265984:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827476.674427:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790266112:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827476.674450:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827476.674469:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827476/real 1787827476] req@ffff9e680a2d7c00 x1874668790266112/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827476.674485:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790266112/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827476.674496:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827476.674498:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827476.674500:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790266112:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827476.674516:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790266240:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827476.674531:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827476.674538:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827476/real 1787827476] req@ffff9e680a2d52c0 x1874668790266240/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827476.674545:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790266240/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827476.674550:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827476.674552:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827476.674554:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790266240:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827476.674557:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790266368:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827476.674566:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827476.674571:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827476/real 1787827476] req@ffff9e680a2d7480 x1874668790266368/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827476.674578:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790266368/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827476.674590:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827476.674592:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827476.674595:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790266368:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827476.674599:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790266496:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827476.674610:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827476.674616:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827476/real 1787827476] req@ffff9e680a2d4b40 x1874668790266496/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827476.674623:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790266496/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827476.674628:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827476.674630:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827476.674632:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790266496:192.168.203.109@tcp:8:kworker.0 00010000:00010000:1.0:1787827478.401836:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827481.794047:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790266624:192.168.203.109@tcp:250:kworker.0 00000800:00000100:0.0:1787827481.794115:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827481.794157:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827481/real 1787827481] req@ffff9e680a2d5a40 x1874668790266624/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827481.794177:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790266624/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827481.794187:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827481.794194:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827481.794228:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790266624:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827481.794278:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790266752:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827481.794311:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827481.794330:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827481/real 1787827481] req@ffff9e680a2d4000 x1874668790266752/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827481.794347:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790266752/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827481.794366:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827481.794375:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827481.794380:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790266752:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827481.794391:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790266880:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827481.794414:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827481.794429:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827481/real 1787827481] req@ffff9e680a2d61c0 x1874668790266880/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827481.794474:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790266880/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827481.794520:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827481.794529:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827481.794539:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790266880:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827481.794550:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790267008:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827481.794578:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827481.794620:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827481/real 1787827481] req@ffff9e680a2d5e00 x1874668790267008/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827481.794642:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790267008/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827481.794668:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827481.794676:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827481.794698:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790267008:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827481.794730:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790267136:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827481.794772:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827481.794819:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827481/real 1787827481] req@ffff9e680a2d4780 x1874668790267136/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827481.794865:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790267136/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827481.794875:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827481.794879:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827481.794883:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790267136:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827481.794890:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790267264:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827481.794905:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827481.794913:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827481/real 1787827481] req@ffff9e680a2d4f00 x1874668790267264/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827481.794921:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790267264/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827481.794928:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827481.794945:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827481.794999:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790267264:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827481.795005:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790267392:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827481.795022:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827481.795030:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827481/real 1787827481] req@ffff9e680a2d5a40 x1874668790267392/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827481.795038:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790267392/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827481.795044:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827481.795046:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827481.795048:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790267392:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827481.795059:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790267520:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827481.795069:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827481.795075:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827481/real 1787827481] req@ffff9e680a2d4b40 x1874668790267520/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827481.795083:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790267520/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827481.795088:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827481.795090:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827481.795092:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790267520:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827481.795095:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790267648:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827481.795127:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827481.795136:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827481/real 1787827481] req@ffff9e680a2d7480 x1874668790267648/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827481.795144:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790267648/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827481.795151:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827481.795153:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827481.795155:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790267648:192.168.203.109@tcp:8:kworker.0 00010000:00010000:1.0:1787827482.497881:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827486.593874:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827486.913822:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827486.913829:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827486.913833:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827486.913841:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4123 00000100:00080000:0.0:1787827486.913850:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827486.913856:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827486.913893:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790267776/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827486.913914:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827486.913917:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827486.913918:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827486.913921:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4123 00000100:00080000:0.0:1787827486.913923:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827486.913925:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827486.913933:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827486.913942:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790267904/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827486.913950:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827486.913953:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827486.913954:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827486.913960:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4123 00000100:00080000:0.0:1787827486.913962:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827486.913971:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827486.913975:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827486.913982:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790268032/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827486.913989:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827486.913992:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827486.913993:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827486.913995:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4123 00000100:00080000:0.0:1787827486.913997:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827486.914000:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827486.914003:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827486.914011:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790268160/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827486.914022:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827486.914024:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827486.914025:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827486.914027:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4123 00000100:00080000:0.0:1787827486.914031:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827486.914036:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827486.914040:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827486.914050:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790268288/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827486.914057:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827486.914060:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827486.914061:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827486.914063:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4123 00000100:00080000:0.0:1787827486.914066:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827486.914068:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827486.914071:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827486.914080:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790268416/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827486.914087:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827486.914090:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827486.914093:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827486.914099:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4123 00000100:00080000:0.0:1787827486.914103:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827486.914109:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827486.914114:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827486.914123:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790268544/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827486.914136:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827486.914143:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827486.914146:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827486.914153:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4123 00000100:00080000:0.0:1787827486.914158:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827486.914162:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827486.914169:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827486.914192:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790268672/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827486.914203:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827486.914209:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827486.914211:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827486.914214:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4123 00000100:00080000:0.0:1787827486.914216:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827486.914219:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827486.914222:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827486.914228:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790268800/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827490.689877:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827492.033790:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827492.033798:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827492.033802:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827492.033810:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4128 00000100:00080000:1.0:1787827492.033821:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827492.033829:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827492.033876:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790268928/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827492.033898:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827492.033901:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827492.033902:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827492.033906:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4128 00000100:00080000:1.0:1787827492.033909:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827492.033912:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827492.033920:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827492.033930:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790269056/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827492.033939:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827492.033944:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827492.033949:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827492.033952:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4128 00000100:00080000:1.0:1787827492.033954:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827492.033956:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827492.033961:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827492.033973:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790269184/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827492.033995:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827492.033999:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827492.034001:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827492.034006:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4128 00000100:00080000:1.0:1787827492.034010:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827492.034020:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827492.034028:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827492.034042:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790269312/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827492.034058:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827492.034062:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827492.034066:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827492.034073:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4128 00000100:00080000:1.0:1787827492.034079:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827492.034082:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827492.034086:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827492.034100:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790269440/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827492.034116:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827492.034120:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827492.034121:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827492.034125:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4128 00000100:00080000:1.0:1787827492.034127:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827492.034129:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827492.034133:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827492.034160:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790269568/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827492.034169:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827492.034172:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827492.034174:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827492.034181:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4128 00000100:00080000:1.0:1787827492.034188:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827492.034193:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827492.034198:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827492.034210:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790269696/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827492.034233:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827492.034237:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827492.034242:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827492.034249:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4128 00000100:00080000:1.0:1787827492.034253:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827492.034260:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827492.034264:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827492.034280:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790269824/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827492.034289:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827492.034294:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827492.034296:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827492.034302:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4128 00000100:00080000:1.0:1787827492.034308:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827492.034314:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827492.034321:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827492.034332:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790269952/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827492.034455:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827492/real 1787827492] req@ffff9e6831aaa940 x1874668790269440/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827492.034463:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790269440/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827492.034468:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827492.034469:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827492.034471:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790269440:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827492.034485:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790269568:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827492.034495:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827492.034502:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827492/real 1787827492] req@ffff9e6831aaa580 x1874668790269568/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827492.034509:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790269568/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827492.034515:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827492.034517:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827492.034519:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790269568:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827492.034522:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790269696:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827492.034535:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827492.034547:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827492/real 1787827492] req@ffff9e6831aa9e00 x1874668790269696/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827492.034558:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790269696/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827492.034573:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827492.034577:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827492.034580:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790269696:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827492.034585:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790269824:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827492.034602:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827492.034613:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827492/real 1787827492] req@ffff9e6831aabc00 x1874668790269824/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827492.034637:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790269824/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827492.034646:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827492.034650:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827492.034653:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790269824:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827492.034659:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790269952:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827492.034675:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827492.034686:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827492/real 1787827492] req@ffff9e6831aa9680 x1874668790269952/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827492.034723:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790269952/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827492.034732:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827492.034735:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827492.034739:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790269952:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827492.545748:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827492.738061:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827493.569822:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:3.0:1787827494.593755:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827494.785865:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827495.617841:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827495.810000:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827496.641774:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827496.833853:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827497.153810:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827497.153824:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827497.153829:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827497.153843:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4133 00000100:00080000:1.0:1787827497.153859:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827497.153869:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827497.153921:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790270080/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827497.153961:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827497.153965:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827497.153967:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827497.153972:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4133 00000100:00080000:1.0:1787827497.153977:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827497.153984:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827497.153995:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827497.154013:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790270208/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827497.154021:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790270080:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827497.154037:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827497.154041:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827497.154043:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827497.154048:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4133 00000100:00080000:1.0:1787827497.154052:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827497.154057:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827497.154062:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827497.154077:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790270336/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827497.154089:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827497.154093:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827497.154094:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827497.154098:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4133 00000100:00080000:1.0:1787827497.154123:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787827497.154124:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827497.154135:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827497.154143:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827497.154158:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790270464/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827497.154170:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827497.154174:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827497.154176:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827497.154184:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4133 00000100:00100000:3.0:1787827497.154184:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827497/real 1787827497] req@ffff9e6831aa8000 x1874668790270080/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827497.154189:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827497.154203:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827497.154214:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827497.154218:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790270080/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827497.154234:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787827497.154241:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790270592/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827497.154243:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827497.154254:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790270080:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827497.154277:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827497.154283:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827497.154287:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827497.154293:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790270208:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827497.154297:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4133 00000100:00080000:1.0:1787827497.154309:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827497.154317:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:3.0:1787827497.154332:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1787827497.154343:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827497.154351:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827497/real 1787827497] req@ffff9e6831aab480 x1874668790270208/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827497.154377:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790270720/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827497.154387:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790270208/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827497.154396:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827497.154406:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827497.154409:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827497.154418:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4133 00000100:00080000:1.0:1787827497.154426:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827497.154436:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827497.154449:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787827497.154454:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787827497.154463:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790270848/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827497.154465:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827497.154479:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790270208:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827497.154486:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827497.154491:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787827497.154495:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790270336:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827497.154505:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827497.154514:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4133 00000100:00080000:1.0:1787827497.154520:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827497.154526:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827497.154533:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787827497.154539:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827497.154555:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790270976/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827497.154562:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827497/real 1787827497] req@ffff9e6831aa9a40 x1874668790270336/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827497.154590:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827497.154595:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787827497.154598:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790270336/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827497.154599:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827497.154607:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4133 00000100:00080000:1.0:1787827497.154610:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827497.154613:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827497.154623:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787827497.154635:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787827497.154643:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790271104/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827497.154648:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827497.154660:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790270336:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827497.154679:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790270464:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827497.154740:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827497.154766:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827497/real 1787827497] req@ffff9e6831aa8b40 x1874668790270464/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827497.154780:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790270464/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827497.154788:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827497.154791:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827497.154795:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790270464:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827497.154801:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790270592:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827497.154830:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827497.154840:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827497/real 1787827497] req@ffff9e6831aa9680 x1874668790270592/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827497.154860:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790270592/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827497.154868:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827497.154871:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827497.154875:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790270592:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827497.154895:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790270720:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827497.154913:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827497.154923:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827497/real 1787827497] req@ffff9e6831aabc00 x1874668790270720/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827497.154934:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790270720/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827497.154943:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827497.154945:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827497.154948:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790270720:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827497.154954:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790270848:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827497.154968:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827497.154978:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827497/real 1787827497] req@ffff9e6831aa9e00 x1874668790270848/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827497.154989:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790270848/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827497.154996:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827497.154999:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827497.155002:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790270848:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827497.155007:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790270976:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827497.155021:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827497.155029:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827497/real 1787827497] req@ffff9e6831aaa580 x1874668790270976/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827497.155040:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790270976/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827497.155048:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827497.155051:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827497.155053:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790270976:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827497.155059:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790271104:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827497.155073:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827497.155081:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827497/real 1787827497] req@ffff9e6831aaa940 x1874668790271104/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827497.155092:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790271104/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827497.155100:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827497.155123:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827497.155127:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790271104:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827497.665784:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:3.0:1787827498.689796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827498.881877:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827499.713752:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827499.905836:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827500.737766:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827500.929878:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827501.761792:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00080000:1.0:1787827502.273816:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827502.273829:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827502.273834:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827502.273846:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4138 00000100:00080000:1.0:1787827502.273858:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827502.273867:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827502.273961:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790271232/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827502.274007:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827502.274012:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827502.274015:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827502.274021:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4138 00000100:00080000:1.0:1787827502.274026:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827502.274031:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827502.274043:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827502.274062:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790271360/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827502.274135:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827502.274137:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790271232:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827502.274149:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827502.274154:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827502.274161:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4138 00000100:00080000:1.0:1787827502.274166:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827502.274182:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827502.274189:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827502.274214:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790271488/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827502.274234:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827502.274240:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827502.274243:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827502.274250:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4138 00000100:00080000:1.0:1787827502.274266:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827502.274274:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827502.274286:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827502.274313:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790271360:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827502.274316:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790271616/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827502.274342:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827502.274347:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827502.274350:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827502.274356:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4138 00000100:00080000:1.0:1787827502.274364:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827502.274369:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790271488:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827502.274377:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827502.274388:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827502.274407:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790271744/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827502.274421:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790271616:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787827502.274422:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827502.274448:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827502.274451:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827502.274458:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4138 00000100:00080000:1.0:1787827502.274463:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827502.274468:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827502.274475:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827502.274477:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790271744:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827502.274500:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790271872/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827502.274525:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790271872:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827502.274532:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827502.274538:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827502.274541:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827502.274548:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4138 00000100:00080000:1.0:1787827502.274557:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827502.274562:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827502.274569:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827502.274587:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790272000/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827502.274609:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827502.274614:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827502.274616:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827502.274631:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4138 00000100:00080000:1.0:1787827502.274635:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827502.274639:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827502.274644:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827502.274657:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790272128/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827502.274671:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790272000:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827502.274672:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827502.274741:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827502.274746:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827502.274767:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4138 00000100:00080000:1.0:1787827502.274776:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827502.274784:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790272128:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787827502.274792:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827502.274808:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827502.274845:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790272256/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827502.274924:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790272256:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827502.785803:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827502.977882:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827503.809808:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827504.001863:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827504.833818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827505.025881:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787827505.346100:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:0.0:1787827505.346167:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787827505.346191:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827505.346245:0:2387:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:0.0:1787827505.346262:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827505.346273:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827505.346282:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827505.346291:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827505.346299:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827505.346309:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827505.346318:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:3.0:1787827505.347186:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827502/real 1787827505] req@ffff9e6831aa9680 x1874668790272256/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827505.347223:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790272256/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827505.347239:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827505.347246:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827505.347252:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790272256:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827505.347263:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827502/real 1787827505] req@ffff9e6831aabc00 x1874668790272128/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827505.347277:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790272128/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827505.347287:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827505.347291:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827505.347295:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790272128:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827505.347303:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827502/real 1787827505] req@ffff9e6831aa9e00 x1874668790272000/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827505.347319:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790272000/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827505.347327:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827505.347332:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827505.347336:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790272000:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827505.347344:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827502/real 1787827505] req@ffff9e6831aaa580 x1874668790271872/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827505.347357:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790271872/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827505.347366:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827505.347371:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827505.347375:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790271872:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827505.347382:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827502/real 1787827505] req@ffff9e6831aaa940 x1874668790271744/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827505.347397:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790271744/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827505.347405:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827505.347409:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827505.347412:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790271744:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827505.347419:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827502/real 1787827505] req@ffff9e6831aa8f00 x1874668790271616/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827505.347430:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790271616/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827505.347438:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827505.347442:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827505.347445:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790271616:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827505.347452:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827502/real 1787827505] req@ffff9e6831aa8780 x1874668790271360/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827505.347464:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790271360/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827505.347487:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827505.347494:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827505.347498:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790271360:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827505.347507:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827502/real 1787827505] req@ffff9e6831aaa1c0 x1874668790271488/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827505.347521:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790271488/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827505.347530:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827505.347534:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827505.347538:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790271488:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827505.347548:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827502/real 1787827505] req@ffff9e6831aaad00 x1874668790271232/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827505.347563:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790271232/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827505.347571:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827505.347634:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827505.347642:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790271232:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787827505.857802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:3.0:1787827506.881808:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827507.073920:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827507.394053:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790272384:192.168.203.109@tcp:250:kworker.0 00000800:00000100:3.0:1787827507.394125:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827507.394174:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827507/real 1787827507] req@ffff9e6804420f00 x1874668790272384/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827507.394202:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790272384/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827507.394226:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827507.394236:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827507.394245:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790272384:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787827507.394260:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790272512:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827507.394295:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827507.394314:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827507/real 1787827507] req@ffff9e6804422d00 x1874668790272512/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827507.394341:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790272512/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827507.394359:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827507.394365:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827507.394372:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790272512:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827507.394406:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790272640:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827507.394441:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827507.394459:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827507/real 1787827507] req@ffff9e6804421e00 x1874668790272640/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827507.394488:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790272640/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827507.394506:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827507.394512:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827507.394519:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790272640:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827507.394531:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790272768:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827507.394574:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827507.394677:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827507/real 1787827507] req@ffff9e6804422940 x1874668790272768/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827507.394721:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790272768/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827507.394733:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827507.394736:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827507.394740:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790272768:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827507.394747:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790272896:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827507.394772:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827507.394786:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827507/real 1787827507] req@ffff9e6804420780 x1874668790272896/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827507.394800:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790272896/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827507.394839:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827507.394844:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827507.394848:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790272896:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827507.394854:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790273024:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827507.394882:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827507.394894:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827507/real 1787827507] req@ffff9e6804421680 x1874668790273024/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827507.394908:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790273024/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827507.394919:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827507.394922:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827507.394926:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790273024:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827507.394933:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790273152:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827507.394951:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827507.394962:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827507/real 1787827507] req@ffff9e68044212c0 x1874668790273152/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827507.394976:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790273152/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827507.394992:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827507.394995:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827507.394999:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790273152:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827507.395021:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790273280:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827507.395057:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827507.395070:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827507/real 1787827507] req@ffff9e6804423840 x1874668790273280/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827507.395083:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790273280/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827507.395092:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827507.395096:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827507.395100:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790273280:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827507.395108:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790273408:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827507.395127:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827507.395139:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827507/real 1787827507] req@ffff9e68044221c0 x1874668790273408/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827507.395153:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790273408/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827507.395161:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827507.395165:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827507.395169:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790273408:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827507.905789:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827508.097859:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827508.929847:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827509.121925:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827509.953774:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:3.0:1787827510.977788:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827511.169873:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827512.001758:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827512.193833:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827512.513802:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827512.513813:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827512.513817:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827512.513827:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4149 00000100:00080000:1.0:1787827512.513838:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827512.513846:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827512.513883:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790273536/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827512.513914:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827512.513918:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827512.513920:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827512.513925:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4149 00000100:00080000:1.0:1787827512.513930:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827512.513935:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827512.513947:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827512.513971:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790273664/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827512.513985:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827512.513989:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827512.513991:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827512.514013:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4149 00000100:00080000:1.0:1787827512.514019:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827512.514028:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827512.514036:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827512.514052:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790273792/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827512.514065:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827512.514075:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827512.514079:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827512.514087:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4149 00000100:00080000:1.0:1787827512.514094:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827512.514104:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827512.514114:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827512.514134:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790273920/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827512.514156:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827512.514160:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827512.514164:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827512.514171:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4149 00000100:00080000:1.0:1787827512.514178:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827512.514183:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827512.514190:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827512.514209:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790274048/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827512.514239:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827512.514248:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827512.514253:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827512.514264:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4149 00000100:00080000:1.0:1787827512.514272:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827512.514280:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827512.514288:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827512.514304:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790274176/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827512.514321:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827512.514325:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827512.514328:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827512.514337:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4149 00000100:00080000:1.0:1787827512.514346:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827512.514353:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827512.514360:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827512.514379:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790274304/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827512.514401:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827512.514405:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827512.514407:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827512.514413:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4149 00000100:00080000:1.0:1787827512.514421:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827512.514428:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827512.514434:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827512.514451:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790274432/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827512.514500:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827512.514508:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827512.514513:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827512.514521:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4149 00000100:00080000:1.0:1787827512.514528:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827512.514535:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827512.514541:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827512.514558:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790274560/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827513.025785:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827513.217836:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827514.049774:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:3.0:1787827515.073877:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827515.266012:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827516.097745:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827516.289909:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827517.121803:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827517.313959:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827517.633753:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827517.633762:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827517.633768:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827517.633780:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4154 00000100:00080000:1.0:1787827517.633791:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827517.633800:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827517.633841:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790274688/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827517.633868:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827517.633872:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827517.633874:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827517.633877:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4154 00000100:00080000:1.0:1787827517.633882:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827517.633889:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827517.633901:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827517.633924:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790274816/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827517.633942:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827517.633946:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827517.633948:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827517.633952:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4154 00000100:00080000:1.0:1787827517.633980:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827517.633988:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827517.633997:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827517.634010:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790274944/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827517.634019:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827517.634022:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827517.634024:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827517.634028:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4154 00000100:00080000:1.0:1787827517.634032:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827517.634040:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827517.634049:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827517.634064:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790275072/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827517.634100:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827517.634108:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827517.634111:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827517.634118:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4154 00000100:00080000:1.0:1787827517.634125:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827517.634130:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827517.634133:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827517.634141:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790275200/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827517.634150:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827517.634155:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827517.634158:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827517.634167:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4154 00000100:00080000:1.0:1787827517.634171:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827517.634175:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827517.634179:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827517.634193:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790275328/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827517.634202:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827517.634206:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827517.634209:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827517.634219:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4154 00000100:00080000:1.0:1787827517.634226:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827517.634232:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827517.634239:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827517.634254:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790275456/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827517.634273:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827517.634281:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827517.634283:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827517.634291:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4154 00000100:00080000:1.0:1787827517.634295:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827517.634298:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827517.634303:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827517.634316:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790275584/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827517.634330:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827517.634339:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827517.634342:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827517.634348:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4154 00000100:00080000:1.0:1787827517.634354:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827517.634364:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827517.634373:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827517.634388:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790275712/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827518.145779:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:3.0:1787827519.169768:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827519.361845:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827520.193912:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827520.385876:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827521.217847:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827521.409898:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827522.241798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00000100:00100000:3.0:1787827522.753984:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790275840:192.168.203.109@tcp:250:kworker.0 00000800:00000100:3.0:1787827522.754081:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827522.754123:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827522/real 1787827522] req@ffff9e6804420b40 x1874668790275840/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827522.754166:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790275840/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827522.754203:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827522.754215:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827522.754231:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790275840:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787827522.754274:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790275968:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827522.754319:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787827522.754336:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4159 00000100:00080000:2.0:1787827522.754339:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827522.754344:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827522/real 1787827522] req@ffff9e6804422580 x1874668790275968/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827522.754346:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787827522.754353:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827522.754370:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790276736/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827522.754380:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790275968/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827522.754382:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827522.754398:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827522.754405:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827522.754416:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4159 00000100:00080000:3.0:1787827522.754419:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827522.754420:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827522.754426:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787827522.754428:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787827522.754435:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827522.754436:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790275968:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827522.754454:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790276864/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827522.754461:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790276096:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827522.754490:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827522.754499:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827522/real 1787827522] req@ffff9e68044230c0 x1874668790276096/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827522.754508:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790276096/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827522.754513:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827522.754516:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827522.754518:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790276096:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827522.754523:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790276224:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827522.754535:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827522.754541:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827522/real 1787827522] req@ffff9e6804423480 x1874668790276224/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827522.754549:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790276224/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827522.754554:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827522.754556:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827522.754558:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790276224:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827522.754562:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790276352:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827522.754572:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827522.754578:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827522/real 1787827522] req@ffff9e68044221c0 x1874668790276352/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827522.754585:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790276352/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827522.754590:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827522.754592:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827522.754594:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790276352:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827522.754598:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790276480:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827522.754609:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827522.754615:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827522/real 1787827522] req@ffff9e6804423840 x1874668790276480/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827522.754623:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790276480/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827522.754629:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827522.754631:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827522.754632:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790276480:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827522.754654:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790276608:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827522.754667:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827522.754674:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827522/real 1787827522] req@ffff9e68044212c0 x1874668790276608/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827522.754681:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790276608/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827522.754687:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827522.754689:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827522.754691:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790276608:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827522.754717:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790276736:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827522.754730:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827522.754737:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827522/real 1787827522] req@ffff9e6804421680 x1874668790276736/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827522.754744:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790276736/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827522.754754:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827522.754756:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827522.754759:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790276736:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827522.754762:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790276864:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827522.754773:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827522.754778:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827522/real 1787827522] req@ffff9e6804420780 x1874668790276864/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827522.754785:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790276864/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827522.754790:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827522.754792:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827522.754794:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790276864:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827523.265764:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827523.457896:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827524.289848:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827524.481976:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827525.313776:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827525.505869:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827526.337786:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827526.529841:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827527.361758:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827527.553863:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827527.873839:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827527.873847:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827527.873851:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827527.873859:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4164 00000100:00080000:0.0:1787827527.873868:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827527.873874:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827527.873925:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790276992/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827527.873957:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827527.873960:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827527.873961:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827527.873964:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4164 00000100:00080000:0.0:1787827527.873966:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827527.873969:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827527.873977:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827527.873987:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790277120/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827527.873994:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827527.873997:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827527.873999:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827527.874002:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4164 00000100:00080000:0.0:1787827527.874004:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827527.874009:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827527.874021:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790276992:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1787827527.874022:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827527.874031:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790277248/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827527.874038:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827527.874040:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827527.874041:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827527.874043:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4164 00000100:00080000:0.0:1787827527.874045:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827527.874047:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827527.874050:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827527.874058:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790277376/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827527.874063:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827527.874065:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827527.874067:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827527.874069:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4164 00000100:00080000:0.0:1787827527.874071:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827527.874074:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827527.874087:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827527.874097:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790277504/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827527.874107:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827527.874111:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:3.0:1787827527.874114:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827527.874115:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827527.874122:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4164 00000100:00080000:0.0:1787827527.874127:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827527.874133:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827527.874138:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827527.874151:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790277632/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827527.874158:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827527.874162:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827527.874164:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827527.874168:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827527/real 1787827527] req@ffff9e6803e99680 x1874668790276992/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827527.874172:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4164 00000100:00080000:0.0:1787827527.874181:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827527.874189:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827527.874198:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827527.874207:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790276992/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827527.874224:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787827527.874225:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790277760/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827527.874234:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787827527.874238:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827527.874242:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790276992:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787827527.874243:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827527.874247:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827527.874255:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4164 00000100:00080000:0.0:1787827527.874258:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827527.874261:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790277120:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827527.874263:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827527.874270:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827527.874280:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790277888/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827527.874286:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827527.874288:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827527.874289:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827527.874299:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4164 00000100:00080000:0.0:1787827527.874303:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827527.874307:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:3.0:1787827527.874312:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787827527.874313:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827527.874324:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790278016/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827527.874332:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827527/real 1787827527] req@ffff9e6803e99e00 x1874668790277120/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827527.874346:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790277120/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827527.874356:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827527.874360:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827527.874364:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790277120:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827527.874388:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790277248:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827527.874409:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827527.874419:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827527/real 1787827527] req@ffff9e6803e98f00 x1874668790277248/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827527.874430:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790277248/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827527.874444:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827527.874447:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827527.874450:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790277248:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827527.874455:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790277376:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827527.874471:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827527.874480:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827527/real 1787827527] req@ffff9e6803e98780 x1874668790277376/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827527.874490:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790277376/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827527.874497:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827527.874500:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827527.874503:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790277376:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827527.874509:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790277504:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827527.874524:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827527.874534:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827527/real 1787827527] req@ffff9e6803e9a580 x1874668790277504/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827527.874546:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790277504/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827527.874553:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827527.874556:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827527.874559:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790277504:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827527.874565:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790277632:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827527.874581:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827527.874590:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827527/real 1787827527] req@ffff9e6803e9a1c0 x1874668790277632/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827527.874601:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790277632/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827527.874610:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827527.874613:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827527.874617:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790277632:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827527.874622:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790277760:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827527.874639:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827527.874650:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827527/real 1787827527] req@ffff9e6803e9b840 x1874668790277760/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827527.874662:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790277760/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827527.874670:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827527.874674:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827527.874677:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790277760:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827527.874683:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790277888:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827527.874748:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827527.874772:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827527/real 1787827527] req@ffff9e6803e9a940 x1874668790277888/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827527.874786:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790277888/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827527.874794:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827527.874798:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827527.874801:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790277888:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827527.874807:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790278016:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827527.874839:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827527.874851:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827527/real 1787827527] req@ffff9e6803e9b480 x1874668790278016/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827527.874864:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790278016/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827527.874872:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827527.874875:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827527.874878:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790278016:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827528.385774:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827528.577908:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827529.409751:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827529.601864:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827530.433905:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827530.626797:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827531.457786:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827531.649920:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827532.481819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827532.673941:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827532.993832:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827532.993850:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827532.993870:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827532.993914:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4169 00000100:00080000:0.0:1787827532.993936:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827532.993946:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827532.994031:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790278144/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827532.994119:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827532.994126:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827532.994128:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827532.994134:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4169 00000100:00080000:0.0:1787827532.994139:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827532.994144:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827532.994158:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827532.994198:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790278144:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827532.994199:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790278272/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827532.994227:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827532.994233:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827532.994236:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827532.994259:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4169 00000100:00080000:0.0:1787827532.994263:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827532.994266:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827532.994271:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827532.994283:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790278400/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827532.994293:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827532.994297:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827532.994299:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827532.994303:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4169 00000100:00080000:0.0:1787827532.994335:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827532.994344:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827532.994350:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827532.994380:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790278528/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827532.994380:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827532.994398:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827532.994404:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827532.994407:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827532.994414:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4169 00000100:00080000:0.0:1787827532.994420:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827532.994427:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827532.994434:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827532.994453:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790278656/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827532.994459:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827532/real 1787827532] req@ffff9e6803e98b40 x1874668790278144/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827532.994476:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827532.994485:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827532.994487:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827532.994496:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4169 00000100:00100000:3.0:1787827532.994498:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790278144/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827532.994505:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827532.994520:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:3.0:1787827532.994520:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787827532.994534:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787827532.994542:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827532.994548:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790278144:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827532.994555:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790278784/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827532.994572:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827532.994576:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827532.994578:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827532.994586:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790278272:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827532.994587:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4169 00000100:00080000:0.0:1787827532.994595:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827532.994618:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827532.994624:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827532.994641:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790278912/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827532.994651:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827532.994668:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827532.994674:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827532/real 1787827532] req@ffff9e6803e9bc00 x1874668790278272/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827532.994678:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827532.994686:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827532.994699:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4169 00000100:00100000:3.0:1787827532.994734:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790278272/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827532.994746:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827532.994755:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787827532.994756:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827532.994763:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790278272:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827532.994770:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827532.994777:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790278400:192.168.203.109@tcp:38:kworker.0 00000008:00100000:0.0:1787827532.994780:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827532.994804:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790279040/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827532.994812:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827532.994827:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827532.994830:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827532/real 1787827532] req@ffff9e6803e992c0 x1874668790278400/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827532.994842:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827532.994849:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827532.994885:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4169 00000100:00100000:3.0:1787827532.994893:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790278400/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827532.994903:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827532.994917:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:3.0:1787827532.994917:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827532.994929:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:0.0:1787827532.994930:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827532.994935:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790278400:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827532.994946:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790278528:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827532.994953:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790279168/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827532.995001:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827532.995033:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827532/real 1787827532] req@ffff9e6803e9ad00 x1874668790278528/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827532.995048:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790278528/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827532.995057:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827532.995061:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827532.995065:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790278528:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827532.995072:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790278656:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827532.995108:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827532.995117:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827532/real 1787827532] req@ffff9e6803e9b480 x1874668790278656/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827532.995128:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790278656/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827532.995142:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827532.995145:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827532.995148:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790278656:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827532.995152:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790278784:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827532.995167:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827532.995175:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827532/real 1787827532] req@ffff9e6803e9a940 x1874668790278784/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827532.995184:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790278784/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827532.995194:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827532.995196:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827532.995199:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790278784:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827532.995218:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790278912:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827532.995232:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827532.995241:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827532/real 1787827532] req@ffff9e6803e9b840 x1874668790278912/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827532.995251:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790278912/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827532.995258:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827532.995261:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827532.995265:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790278912:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827532.995270:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790279040:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827532.995300:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827532.995309:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827532/real 1787827532] req@ffff9e6803e9a1c0 x1874668790279040/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827532.995326:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790279040/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827532.995333:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827532.995335:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827532.995338:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790279040:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827532.995342:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790279168:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827532.995356:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827532.995364:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827532/real 1787827532] req@ffff9e6803e9a580 x1874668790279168/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827532.995380:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790279168/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827532.995390:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827532.995407:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827532.995412:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790279168:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827533.505756:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827533.697922:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827534.529791:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827534.721832:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827535.553761:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827535.745871:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827536.577801:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827536.769866:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827537.601780:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827537.793931:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827538.113831:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827538.113839:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827538.113843:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827538.113879:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4174 00000100:00080000:2.0:1787827538.113890:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827538.113896:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827538.113942:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790279296/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827538.113967:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827538.113971:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827538.113973:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827538.113987:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4174 00000100:00080000:2.0:1787827538.113990:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827538.113993:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787827538.114001:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827538.114015:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790279424/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827538.114021:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790279296:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787827538.114030:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827538.114035:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827538.114037:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827538.114053:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4174 00000100:00080000:2.0:1787827538.114072:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827538.114075:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787827538.114078:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827538.114087:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790279552/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827538.114094:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827538.114097:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827538.114098:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827538.114100:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4174 00000100:00080000:2.0:1787827538.114104:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827538.114107:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787827538.114111:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827538.114120:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790279680/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827538.114121:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787827538.114160:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827538.114164:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827538.114166:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827538.114170:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4174 00000100:00080000:2.0:1787827538.114173:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827538.114177:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787827538.114181:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827538.114191:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790279808/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827538.114199:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827538.114201:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827538.114203:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827538.114214:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4174 00000100:00080000:2.0:1787827538.114217:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827538.114223:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827538/real 1787827538] req@ffff9e6804422940 x1874668790279296/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827538.114225:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787827538.114231:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827538.114241:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790279936/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827538.114272:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827538.114274:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790279296/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827538.114276:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827538.114281:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827538.114307:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4174 00000100:00080000:2.0:1787827538.114312:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827538.114313:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827538.114317:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:3.0:1787827538.114322:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:2.0:1787827538.114324:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827538.114328:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790279296:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787827538.114334:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790280064/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827538.114341:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827538.114344:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827538.114345:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827538.114347:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4174 00000100:00080000:2.0:1787827538.114350:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827538.114370:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790279424:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787827538.114371:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787827538.114380:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827538.114394:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790280192/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827538.114404:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827538.114407:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827538.114413:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787827538.114413:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787827538.114423:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4174 00000100:00080000:2.0:1787827538.114429:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827538.114437:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827538.114437:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827538/real 1787827538] req@ffff9e6804421e00 x1874668790279424/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787827538.114441:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827538.114457:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790279424/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827538.114467:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787827538.114471:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790280320/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827538.114472:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827538.114481:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790279424:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827538.114514:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790279552:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827538.114533:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827538.114540:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827538/real 1787827538] req@ffff9e6804422d00 x1874668790279552/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827538.114548:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790279552/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827538.114554:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827538.114556:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827538.114558:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790279552:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827538.114579:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790279680:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827538.114594:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827538.114602:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827538/real 1787827538] req@ffff9e6804420f00 x1874668790279680/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827538.114617:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790279680/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827538.114628:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827538.114632:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827538.114636:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790279680:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827538.114644:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790279808:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827538.114672:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827538.114687:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827538/real 1787827538] req@ffff9e6804420780 x1874668790279808/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827538.114699:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790279808/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827538.114706:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827538.114737:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827538.114739:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790279808:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827538.114743:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790279936:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827538.114758:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827538.114765:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827538/real 1787827538] req@ffff9e6804421680 x1874668790279936/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827538.114795:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790279936/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827538.114802:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827538.114804:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827538.114831:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790279936:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827538.114837:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790280064:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827538.114854:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827538.114878:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827538/real 1787827538] req@ffff9e68044212c0 x1874668790280064/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827538.114899:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790280064/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827538.114910:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827538.114912:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827538.114915:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790280064:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827538.114931:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790280192:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827538.114945:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827538.114951:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827538/real 1787827538] req@ffff9e6804423840 x1874668790280192/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827538.114959:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790280192/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827538.114965:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827538.114967:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827538.114969:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790280192:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827538.114974:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790280320:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827538.114994:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827538.115000:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827538/real 1787827538] req@ffff9e68044221c0 x1874668790280320/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827538.115008:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790280320/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827538.115013:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827538.115016:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827538.115018:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790280320:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827538.625842:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827538.817886:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827539.649786:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827539.842019:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827540.673884:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827540.866002:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827541.697874:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827541.889922:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827542.721748:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827542.913891:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827543.233884:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827543.233913:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827543.233918:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827543.233930:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4179 00000100:00080000:0.0:1787827543.233944:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827543.233953:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827543.233997:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790280448/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827543.234060:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827543.234066:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827543.234070:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827543.234077:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4179 00000100:00080000:0.0:1787827543.234084:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827543.234092:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827543.234107:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827543.234126:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790280448:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827543.234129:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790280576/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827543.234147:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827543.234155:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827543.234158:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827543.234164:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4179 00000100:00080000:0.0:1787827543.234170:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827543.234176:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827543.234183:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827543.234197:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790280704/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827543.234210:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827543.234213:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827543.234218:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827543.234246:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827543.234252:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4179 00000100:00080000:0.0:1787827543.234260:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827543.234266:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827543.234274:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827543.234292:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790280832/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827543.234292:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827543/real 1787827543] req@ffff9e6803e98780 x1874668790280448/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827543.234326:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827543.234331:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827543.234334:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827543.234339:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790280448/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827543.234340:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4179 00000100:00080000:0.0:1787827543.234346:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827543.234352:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:3.0:1787827543.234357:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787827543.234360:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827543.234377:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790280960/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827543.234382:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827543.234390:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790280448:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787827543.234395:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827543.234400:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827543.234419:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827543.234425:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4179 00000100:00080000:0.0:1787827543.234432:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827543.234438:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827543.234442:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790280576:192.168.203.109@tcp:38:kworker.0 00000008:00100000:0.0:1787827543.234447:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827543.234464:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790281088/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827543.234482:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787827543.234483:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827543.234488:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827543.234492:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827543.234499:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4179 00000100:00080000:0.0:1787827543.234506:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827543.234506:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827543/real 1787827543] req@ffff9e6803e98f00 x1874668790280576/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827543.234545:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790280576/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827543.234548:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827543.234557:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787827543.234561:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827543.234568:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827543.234575:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790281216/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827543.234575:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790280576:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827543.234586:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790280704:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827543.234593:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827543.234598:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827543.234602:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827543.234625:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4179 00000100:00080000:0.0:1787827543.234631:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787827543.234636:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827543.234650:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827543.234658:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827543.234667:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827543/real 1787827543] req@ffff9e6803e99e00 x1874668790280704/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827543.234675:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790281344/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827543.234691:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790280704/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827543.234693:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827543.234701:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827543.234704:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827543.234708:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827543.234762:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4179 00000100:00080000:0.0:1787827543.234771:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827543.234779:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787827543.234780:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827543.234842:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827543.234842:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790280704:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827543.234855:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790280832:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827543.234863:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790281472/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827543.234914:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827543.234938:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827543/real 1787827543] req@ffff9e6803e99680 x1874668790280832/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827543.234962:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790280832/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827543.234979:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827543.234985:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827543.234990:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790280832:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827543.234999:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790280960:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827543.235054:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827543.235068:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827543/real 1787827543] req@ffff9e6803e9a580 x1874668790280960/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827543.235081:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790280960/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827543.235091:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827543.235095:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827543.235098:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790280960:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827543.235120:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790281088:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827543.235139:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827543.235150:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827543/real 1787827543] req@ffff9e6803e9a1c0 x1874668790281088/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827543.235163:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790281088/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827543.235174:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827543.235177:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827543.235180:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790281088:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827543.235186:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790281216:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827543.235220:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827543.235233:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827543/real 1787827543] req@ffff9e6803e9b840 x1874668790281216/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827543.235256:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790281216/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827543.235266:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827543.235270:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827543.235273:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790281216:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827543.235279:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790281344:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827543.235298:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827543.235309:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827543/real 1787827543] req@ffff9e6803e9a940 x1874668790281344/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827543.235321:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790281344/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827543.235331:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827543.235351:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827543.235354:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790281344:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827543.235361:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790281472:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827543.235379:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827543.235389:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827543/real 1787827543] req@ffff9e6803e9b480 x1874668790281472/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827543.235419:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790281472/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827543.235436:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827543.235440:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827543.235444:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790281472:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827543.745851:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827543.937995:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827544.769792:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827544.961846:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827545.793799:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827545.985983:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827546.817779:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827547.009837:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827547.841745:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827548.033893:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827548.353840:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827548.353851:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827548.353856:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827548.353868:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4185 00000100:00080000:3.0:1787827548.353879:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827548.353887:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827548.353922:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790281600/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827548.353950:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827548.353955:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827548.353958:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827548.353964:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4185 00000100:00080000:3.0:1787827548.353970:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827548.353975:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827548.353987:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827548.353990:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790281600:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787827548.354004:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790281728/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827548.354018:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827548.354022:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827548.354025:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827548.354030:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4185 00000100:00080000:3.0:1787827548.354037:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827548.354042:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827548.354047:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827548.354053:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827548.354062:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790281856/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827548.354075:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827548.354079:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787827548.354094:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827548/real 1787827548] req@ffff9e680a2d52c0 x1874668790281600/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827548.354110:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790281600/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827548.354117:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827548.354120:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827548.354123:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790281600:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787827548.354142:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790281728:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827548.354161:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827548.354170:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827548.354171:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827548/real 1787827548] req@ffff9e680a2d7c00 x1874668790281728/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827548.354180:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4185 00000100:00080000:3.0:1787827548.354190:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827548.354198:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790281728/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827548.354199:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827548.354210:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787827548.354225:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827548.354228:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827548.354231:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790281728:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827548.354236:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790281856:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827548.354253:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827548.354262:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827548/real 1787827548] req@ffff9e680a2d70c0 x1874668790281856/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827548.354270:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790281856/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827548.354275:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827548.354277:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827548.354279:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790281856:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827548.354353:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790281984/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827548.354406:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790281984:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787827548.354420:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827548.354426:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827548.354429:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787827548.354436:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827548.354437:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4185 00000100:00080000:3.0:1787827548.354446:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827548.354452:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827548.354458:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827548/real 1787827548] req@ffff9e680a2d43c0 x1874668790281984/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787827548.354461:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827548.354484:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790282112/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827548.354485:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790281984/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827548.354506:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827548.354512:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827548.354516:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827548.354517:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790281984:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787827548.354526:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827548.354529:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827548.354534:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790282112:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787827548.354538:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4185 00000100:00080000:3.0:1787827548.354544:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827548.354549:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827548.354555:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827548.354556:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827548.354567:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827548/real 1787827548] req@ffff9e680a2d6580 x1874668790282112/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827548.354575:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790282240/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827548.354581:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790282112/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827548.354603:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827548.354604:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827548.354608:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827548.354612:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787827548.354613:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790282112:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787827548.354618:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827548.354626:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790282240:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827548.354626:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4185 00000100:00080000:3.0:1787827548.354632:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827548.354640:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787827548.354644:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787827548.354646:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827548.354654:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827548/real 1787827548] req@ffff9e680a2d6d00 x1874668790282240/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827548.354663:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790282368/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827548.354665:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790282240/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827548.354713:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827548.354717:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827548.354720:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790282240:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827548.354740:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827548.354744:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787827548.354757:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790282368:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827548.354761:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827548.354769:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4185 00000100:00080000:3.0:1787827548.354776:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827548.354780:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827548.354785:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827548.354792:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827548.354794:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827548/real 1787827548] req@ffff9e680a2d6580 x1874668790282368/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827548.354805:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790282368/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827548.354811:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790282496/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827548.354813:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827548.354820:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827548.354827:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790282368:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827548.354840:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827548.354844:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790282496:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787827548.354847:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827548.354851:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827548.354857:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4185 00000800:00000100:2.0:1787827548.354861:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827548.354863:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827548.354871:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827548.354872:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827548/real 1787827548] req@ffff9e680a2d43c0 x1874668790282496/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787827548.354881:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827548.354885:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790282496/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827548.354893:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827548.354896:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827548.354901:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790282624/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827548.354902:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790282496:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827548.354935:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790282624:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827548.354954:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827548.354966:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827548/real 1787827548] req@ffff9e680a2d70c0 x1874668790282624/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827548.354979:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790282624/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827548.354989:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827548.354992:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827548.354996:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790282624:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827548.865803:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827549.057864:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827549.889825:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827550.081857:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827550.913779:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827551.105874:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827551.937806:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827552.129854:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827552.961804:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827553.153974:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827553.473784:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827553.473792:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827553.473796:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827553.473804:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4190 00000100:00080000:3.0:1787827553.473813:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827553.473820:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827553.473852:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790282752/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827553.473875:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827553.473878:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827553.473879:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827553.473882:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4190 00000100:00080000:3.0:1787827553.473885:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827553.473888:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827553.473896:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827553.473906:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790282880/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827553.473910:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790282752:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827553.473921:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827553.473925:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827553.473926:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827553.473937:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4190 00000100:00080000:3.0:1787827553.473940:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827553.473943:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827553.473946:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827553.473955:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790283008/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827553.473979:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827553.473981:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827553.473983:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827553.473985:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4190 00000100:00080000:3.0:1787827553.473988:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827553.473991:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827553.473995:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827553.473996:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827553.474010:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790283136/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827553.474017:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827553.474020:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827553.474021:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827553.474023:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4190 00000100:00080000:3.0:1787827553.474025:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827553.474028:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827553.474031:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827553.474037:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827553/real 1787827553] req@ffff9e680a2d7c00 x1874668790282752/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827553.474041:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790283264/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827553.474065:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827553.474066:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790282752/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827553.474069:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827553.474088:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827553.474096:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4190 00000100:00080000:3.0:1787827553.474101:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827553.474102:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827553.474106:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827553.474110:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787827553.474111:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827553.474118:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790282752:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787827553.474120:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790283392/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827553.474131:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827553.474134:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827553.474135:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827553.474137:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4190 00000100:00080000:3.0:1787827553.474140:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827553.474142:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827553.474145:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827553.474168:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790282880:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827553.474178:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790283520/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827553.474186:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827553.474188:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827553.474190:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827553.474193:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4190 00000800:00000100:2.0:1787827553.474197:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827553.474199:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827553.474204:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827553.474208:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827553.474212:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827553/real 1787827553] req@ffff9e680a2d52c0 x1874668790282880/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827553.474235:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790283648/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827553.474242:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827553.474248:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827553.474252:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827553.474257:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4190 00000100:00080000:3.0:1787827553.474260:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827553.474263:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790282880/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827553.474266:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827553.474272:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787827553.474276:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787827553.474298:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790283776/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827553.474303:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827553.474310:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790282880:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827553.474316:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790283008:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827553.474333:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827553.474340:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827553/real 1787827553] req@ffff9e680a2d7480 x1874668790283008/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827553.474366:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790283008/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827553.474375:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827553.474378:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827553.474382:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790283008:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827553.474388:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790283136:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827553.474410:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827553.474422:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827553/real 1787827553] req@ffff9e680a2d4b40 x1874668790283136/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827553.474450:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790283136/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827553.474459:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827553.474462:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827553.474465:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790283136:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827553.474472:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790283264:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827553.474495:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827553.474505:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827553/real 1787827553] req@ffff9e680a2d5a40 x1874668790283264/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827553.474514:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790283264/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827553.474519:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827553.474521:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827553.474523:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790283264:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827553.474527:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790283392:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827553.474540:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827553.474546:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827553/real 1787827553] req@ffff9e680a2d4f00 x1874668790283392/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827553.474553:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790283392/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827553.474559:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827553.474561:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827553.474562:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790283392:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827553.474576:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790283520:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827553.474586:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827553.474592:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827553/real 1787827553] req@ffff9e680a2d4780 x1874668790283520/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827553.474599:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790283520/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827553.474604:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827553.474606:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827553.474608:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790283520:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827553.474612:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790283648:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827553.474638:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827553.474645:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827553/real 1787827553] req@ffff9e680a2d5e00 x1874668790283648/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827553.474652:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790283648/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827553.474657:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827553.474659:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827553.474661:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790283648:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827553.474665:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790283776:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827553.474675:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827553.474680:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827553/real 1787827553] req@ffff9e680a2d61c0 x1874668790283776/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827553.474688:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790283776/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827553.474692:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827553.474694:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827553.474697:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790283776:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827553.985771:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827554.177885:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827555.009787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827555.201857:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827556.033791:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827556.225829:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827557.057860:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827557.249872:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827558.081780:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827558.273848:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827558.593780:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827558.593790:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827558.593794:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827558.593804:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4195 00000100:00080000:3.0:1787827558.593815:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827558.593822:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827558.593857:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790283904/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827558.593884:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827558.593888:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827558.593889:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827558.593893:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4195 00000100:00080000:3.0:1787827558.593904:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827558.593910:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827558.593921:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827558.593932:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790283904:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787827558.593939:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790284032/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827558.593952:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827558.593955:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827558.593957:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827558.593961:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4195 00000100:00080000:3.0:1787827558.593965:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827558.593969:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827558.593975:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827558.593987:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790284160/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827558.593997:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827558.594002:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827558.594003:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827558.594009:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4195 00000800:00000100:2.0:1787827558.594012:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827558.594016:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827558.594031:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827558.594038:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827558.594051:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827558/real 1787827558] req@ffff9e680a2d4000 x1874668790283904/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827558.594054:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790284288/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827558.594076:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827558.594078:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790283904/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827558.594083:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827558.594086:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827558.594091:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4195 00000100:00080000:3.0:1787827558.594100:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827558.594103:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827558.594108:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787827558.594110:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827558.594116:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790283904:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787827558.594118:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827558.594132:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790284416/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827558.594139:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790284032:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827558.594149:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827558.594153:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827558.594156:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827558.594162:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4195 00000100:00080000:3.0:1787827558.594167:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827558.594171:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827558.594175:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827558.594180:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827558.594202:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827558/real 1787827558] req@ffff9e680a2d61c0 x1874668790284032/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827558.594204:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790284544/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827558.594224:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827558.594229:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827558.594232:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827558.594239:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4195 00000100:00080000:3.0:1787827558.594243:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827558.594245:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790284032/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827558.594250:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827558.594261:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787827558.594263:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827558.594268:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827558.594272:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790284032:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827558.594276:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790284672/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827558.594281:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790284160:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827558.594301:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827558.594305:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827558.594307:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827558.594312:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4195 00000800:00000100:2.0:1787827558.594314:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827558.594322:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827558.594329:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827558/real 1787827558] req@ffff9e680a2d5e00 x1874668790284160/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827558.594330:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827558.594342:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827558.594346:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790284160/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827558.594353:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827558.594355:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827558.594357:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790284160:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827558.594362:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790284288:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827558.594368:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790284800/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827558.594377:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827558.594385:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827558.594388:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827558/real 1787827558] req@ffff9e680a2d4780 x1874668790284288/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827558.594393:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827558.594399:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827558.594408:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4195 00000100:00100000:2.0:1787827558.594412:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790284288/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827558.594415:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827558.594425:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787827558.594430:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787827558.594433:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787827558.594438:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827558.594443:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790284288:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827558.594450:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790284928/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827558.594461:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790284416:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827558.594481:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827558.594489:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827558/real 1787827558] req@ffff9e680a2d4f00 x1874668790284416/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827558.594502:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790284416/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827558.594508:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827558.594510:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827558.594512:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790284416:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827558.594515:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790284544:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827558.594525:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827558.594531:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827558/real 1787827558] req@ffff9e680a2d5a40 x1874668790284544/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827558.594538:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790284544/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827558.594544:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827558.594546:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827558.594548:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790284544:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827558.594552:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790284672:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827558.594561:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827558.594567:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827558/real 1787827558] req@ffff9e680a2d4b40 x1874668790284672/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827558.594574:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790284672/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827558.594583:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827558.594585:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827558.594587:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790284672:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827558.594590:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790284800:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827558.594608:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827558.594615:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827558/real 1787827558] req@ffff9e680a2d7480 x1874668790284800/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827558.594622:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790284800/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827558.594627:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827558.594629:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827558.594631:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790284800:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827558.594642:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790284928:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827558.594652:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827558.594658:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827558/real 1787827558] req@ffff9e680a2d52c0 x1874668790284928/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827558.594665:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790284928/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827558.594670:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827558.594672:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827558.594674:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790284928:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827559.105777:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827559.297887:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827560.129787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827560.321848:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827561.153756:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827561.345856:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827562.177806:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827562.369937:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827563.201832:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827563.393944:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827563.713788:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827563.713797:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827563.713801:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827563.713809:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4200 00000100:00080000:0.0:1787827563.713818:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827563.713824:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827563.713865:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790285056/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827563.713905:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827563.713910:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827563.713912:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827563.713916:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4200 00000100:00080000:0.0:1787827563.713919:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827563.713923:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827563.713933:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827563.713946:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790285184/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827563.713955:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790285056:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787827563.713958:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827563.713965:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827563.713966:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827563.713970:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4200 00000100:00080000:0.0:1787827563.713972:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827563.713975:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827563.713978:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827563.713987:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790285312/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827563.713994:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827563.713997:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827563.713998:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827563.714001:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4200 00000100:00080000:0.0:1787827563.714005:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827563.714008:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827563.714012:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827563.714020:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790285440/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827563.714024:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827563.714034:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827563.714038:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827563.714040:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827563.714044:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4200 00000100:00080000:0.0:1787827563.714058:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827563.714061:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827563.714064:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827563.714075:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790285568/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827563.714081:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827563/real 1787827563] req@ffff9e6803e9ad00 x1874668790285056/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827563.714086:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827563.714094:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827563.714098:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827563.714106:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4200 00000100:00080000:0.0:1787827563.714109:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827563.714116:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827563.714116:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790285056/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787827563.714121:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827563.714134:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790285696/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827563.714135:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827563.714147:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827563.714156:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790285056:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787827563.714165:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827563.714170:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827563.714173:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827563.714178:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4200 00000100:00080000:0.0:1787827563.714183:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827563.714184:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790285184:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827563.714193:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827563.714200:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827563.714212:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827563.714223:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827563/real 1787827563] req@ffff9e6803e992c0 x1874668790285184/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827563.714229:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790285824/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827563.714246:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790285184/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827563.714247:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827563.714257:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827563.714261:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827563.714267:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827563.714268:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4200 00000100:00080000:0.0:1787827563.714276:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827563.714277:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787827563.714285:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827563.714286:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790285184:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827563.714294:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790285312:192.168.203.109@tcp:38:kworker.0 00000008:00100000:0.0:1787827563.714297:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827563.714313:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790285952/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827563.714321:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827563.714329:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827563.714333:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827563.714337:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827563.714337:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827563/real 1787827563] req@ffff9e6803e9bc00 x1874668790285312/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827563.714346:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4200 00000100:00080000:0.0:1787827563.714352:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827563.714358:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827563.714362:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827563.714364:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790285312/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827563.714372:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790286080/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827563.714395:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827563.714402:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827563.714408:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790285312:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827563.714418:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790285440:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827563.714444:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827563.714459:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827563/real 1787827563] req@ffff9e6803e98b40 x1874668790285440/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827563.714485:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790285440/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827563.714496:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827563.714500:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827563.714504:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790285440:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827563.714511:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790285568:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827563.714536:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827563.714549:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827563/real 1787827563] req@ffff9e6803e9b480 x1874668790285568/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827563.714564:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790285568/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827563.714570:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827563.714572:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827563.714575:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790285568:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827563.714579:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790285696:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827563.714594:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827563.714601:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827563/real 1787827563] req@ffff9e6803e9a940 x1874668790285696/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827563.714609:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790285696/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827563.714616:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827563.714618:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827563.714620:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790285696:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827563.714635:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790285824:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827563.714668:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827563.714678:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827563/real 1787827563] req@ffff9e6803e9b840 x1874668790285824/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827563.714686:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790285824/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827563.714721:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827563.714723:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827563.714726:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790285824:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827563.714731:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790285952:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827563.714746:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827563.714754:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827563/real 1787827563] req@ffff9e6803e9a1c0 x1874668790285952/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827563.714762:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790285952/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827563.714768:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827563.714771:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827563.714773:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790285952:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827563.714777:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790286080:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827563.714788:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827563.714794:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827563/real 1787827563] req@ffff9e6803e9a580 x1874668790286080/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827563.714802:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790286080/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827563.714807:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827563.714809:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827563.714812:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790286080:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827564.225783:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827564.417883:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827565.249826:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827565.441883:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827566.273791:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827566.465864:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827567.297802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827567.489812:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827568.321783:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827568.514005:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827568.833742:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827568.833750:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827568.833754:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827568.833762:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4205 00000100:00080000:0.0:1787827568.833770:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827568.833775:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827568.833806:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790286208/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827568.833830:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827568.833833:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827568.833834:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827568.833837:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4205 00000100:00080000:0.0:1787827568.833839:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827568.833853:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827568.833860:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827568.833868:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790286336/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827568.833874:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827568.833879:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827568.833880:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827568.833882:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4205 00000100:00080000:0.0:1787827568.833884:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827568.833886:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827568.833889:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827568.833895:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790286464/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827568.833901:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827568.833903:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827568.833904:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827568.833907:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4205 00000100:00080000:0.0:1787827568.833909:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827568.833912:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827568.833916:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827568.833919:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790286208:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827568.833925:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790286592/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827568.833932:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827568.833935:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827568.833936:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827568.833939:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4205 00000100:00080000:0.0:1787827568.833941:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827568.833943:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827568.833947:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827568.833956:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790286720/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827568.833962:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827568.833964:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827568.833966:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827568.833969:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4205 00000100:00080000:0.0:1787827568.833971:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827568.833974:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827568.833977:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827568.833983:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790286848/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827568.833989:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827568.833991:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827568.833993:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827568.833995:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4205 00000100:00080000:0.0:1787827568.833997:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827568.833999:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827568.834002:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827568.834012:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790286976/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827568.834015:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790286336:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827568.834029:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827568.834031:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827568.834035:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827568.834040:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4205 00000100:00080000:0.0:1787827568.834043:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827568.834046:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827568.834049:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827568.834050:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790286464:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827568.834063:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790287104/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827568.834071:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827568.834074:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827568.834077:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827568.834080:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4205 00000100:00080000:0.0:1787827568.834086:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827568.834093:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827568.834094:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790286592:192.168.203.109@tcp:8:kworker.0 00000008:00100000:0.0:1787827568.834097:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827568.834107:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790287232/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827568.834137:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790286720:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827568.834164:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790286848:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827568.834186:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790286976:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827568.834217:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790287104:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827568.834244:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790287232:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827569.345849:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827569.537904:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827570.369819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827570.561898:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827571.393787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827571.585891:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787827571.906103:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:0.0:1787827571.906157:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787827571.906176:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827571.906235:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:0.0:1787827571.906269:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827571.906280:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827571.906293:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787827571.906300:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827568/real 1787827571] req@ffff9e6803e9b480 x1874668790287232/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787827571.906311:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827571.906329:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787827571.906333:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790287232/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787827571.906346:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00080000:2.0:1787827571.906348:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827571.906357:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000800:00000100:0.0:1787827571.906362:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787827571.906368:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790287232:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827571.906381:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827568/real 1787827571] req@ffff9e6803e98780 x1874668790286592/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827571.906397:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790286592/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827571.906406:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827571.906411:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827571.906415:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790286592:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827571.906422:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827568/real 1787827571] req@ffff9e6803e9a580 x1874668790286720/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827571.906436:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790286720/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827571.906446:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827571.906450:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827571.906454:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790286720:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827571.906461:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827568/real 1787827571] req@ffff9e6803e9a1c0 x1874668790286848/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827571.906475:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790286848/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827571.906486:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827571.906490:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827571.906494:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790286848:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827571.906502:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827568/real 1787827571] req@ffff9e6803e9b840 x1874668790286976/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827571.906516:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790286976/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827571.906524:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827571.906528:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827571.906532:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790286976:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827571.906539:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827568/real 1787827571] req@ffff9e6803e9a940 x1874668790287104/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827571.906551:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790287104/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827571.906560:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827571.906564:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827571.906568:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790287104:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827571.906575:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827568/real 1787827571] req@ffff9e6803e99680 x1874668790286208/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827571.906611:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790286208/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827571.906621:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827571.906624:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827571.906629:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790286208:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787827571.906652:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827568/real 1787827571] req@ffff9e6803e99e00 x1874668790286336/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827571.906665:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790286336/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827571.906673:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827571.906677:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827571.906680:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790286336:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827571.906687:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827568/real 1787827571] req@ffff9e6803e98f00 x1874668790286464/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827571.906699:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790286464/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827571.906707:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827571.906744:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827571.906748:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790286464:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827572.417772:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827572.609953:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827573.441746:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827573.633841:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827573.953808:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827573.953816:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827573.953819:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827573.953828:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4210 00000100:00080000:3.0:1787827573.953836:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827573.953842:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827573.953869:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790287360/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827573.953889:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827573.953892:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827573.953893:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827573.953896:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4210 00000100:00080000:3.0:1787827573.953898:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827573.953901:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827573.953908:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827573.953918:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790287488/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827573.953922:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790287360:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827573.953931:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827573.953935:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827573.953936:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827573.953939:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4210 00000100:00080000:3.0:1787827573.953940:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827573.953943:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827573.953946:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827573.953953:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790287616/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827573.953964:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827573.953967:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827573.953968:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827573.953971:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4210 00000100:00080000:3.0:1787827573.953974:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827573.953976:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827573.953979:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827573.953983:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827573.953988:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790287744/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827573.953995:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827573.953998:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827573.954000:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827573.954002:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4210 00000100:00080000:3.0:1787827573.954005:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827573.954009:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827573.954014:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827573.954018:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827573/real 1787827573] req@ffff9e680a2d7c00 x1874668790287360/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827573.954029:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790287872/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827573.954036:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790287360/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827573.954050:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827573.954056:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827573.954057:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827573.954059:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827573.954064:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4210 00000100:00080000:3.0:1787827573.954071:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827573.954075:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827573.954081:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827573.954085:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790287360:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787827573.954089:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827573.954106:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790288000/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827573.954117:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790287488:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827573.954118:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827573.954127:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827573.954129:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827573.954135:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4210 00000100:00080000:3.0:1787827573.954140:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827573.954145:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827573.954151:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827573.954154:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827573.954169:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790288128/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827573.954176:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827573/real 1787827573] req@ffff9e680a2d70c0 x1874668790287488/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827573.954188:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827573.954194:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827573.954204:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827573.954207:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790287488/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827573.954211:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4210 00000100:00080000:3.0:1787827573.954219:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827573.954223:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827573.954226:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787827573.954232:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:3.0:1787827573.954234:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827573.954238:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790287488:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827573.954248:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790287616:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827573.954252:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790288256/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827573.954272:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787827573.954277:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827573.954290:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827573/real 1787827573] req@ffff9e680a2d43c0 x1874668790287616/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827573.954294:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827573.954299:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827573.954307:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4210 00000100:00100000:2.0:1787827573.954310:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790287616/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827573.954314:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827573.954323:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787827573.954325:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787827573.954330:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787827573.954332:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827573.954338:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790287616:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827573.954346:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790287744:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827573.954352:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790288384/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827573.954373:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827573.954388:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827573/real 1787827573] req@ffff9e680a2d6580 x1874668790287744/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827573.954400:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790287744/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827573.954409:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827573.954412:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827573.954415:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790287744:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827573.954420:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790287872:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827573.954440:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827573.954450:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827573/real 1787827573] req@ffff9e680a2d6d00 x1874668790287872/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827573.954461:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790287872/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827573.954469:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827573.954472:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827573.954476:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790287872:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827573.954494:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790288000:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827573.954514:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827573.954526:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827573/real 1787827573] req@ffff9e680a2d52c0 x1874668790288000/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827573.954545:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790288000/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827573.954555:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827573.954558:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827573.954561:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790288000:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827573.954567:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790288128:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827573.954600:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827573.954613:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827573/real 1787827573] req@ffff9e680a2d7480 x1874668790288128/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827573.954626:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790288128/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827573.954633:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827573.954636:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827573.954640:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790288128:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827573.954647:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790288256:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827573.954668:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827573.954687:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827573/real 1787827573] req@ffff9e680a2d4b40 x1874668790288256/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827573.954724:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790288256/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827573.954738:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827573.954741:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827573.954744:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790288256:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827573.954750:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790288384:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827573.954770:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827573.954781:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827573/real 1787827573] req@ffff9e680a2d5a40 x1874668790288384/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827573.954791:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790288384/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827573.954799:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827573.954802:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827573.954805:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790288384:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827574.465755:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827574.657906:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827575.489834:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827575.681799:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827576.513857:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827576.705930:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827577.537796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827577.730030:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827578.561776:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827578.753956:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827579.073752:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827579.073760:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827579.073775:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827579.073783:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4215 00000100:00080000:3.0:1787827579.073792:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827579.073798:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827579.073829:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790288512/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827579.073852:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827579.073855:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827579.073857:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827579.073859:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4215 00000100:00080000:3.0:1787827579.073862:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827579.073864:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827579.073871:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827579.073880:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790288640/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827579.073887:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827579.073890:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827579.073891:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827579.073894:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4215 00000100:00080000:3.0:1787827579.073896:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827579.073899:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790288512:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827579.073901:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827579.073909:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827579.073917:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790288768/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827579.073923:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827579.073925:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827579.073927:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827579.073931:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4215 00000100:00080000:3.0:1787827579.073935:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827579.073939:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827579.073945:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827579.073957:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790288896/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827579.073967:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827579.073972:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827579.073977:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827579.073979:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827579.073985:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4215 00000100:00080000:3.0:1787827579.073989:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827579.073994:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827579.073998:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827579.074008:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790289024/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827579.074014:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827579/real 1787827579] req@ffff9e680a2d4f00 x1874668790288512/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827579.074018:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827579.074023:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827579.074026:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827579.074034:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4215 00000100:00080000:3.0:1787827579.074039:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827579.074042:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827579.074046:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827579.074050:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790288512/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827579.074056:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790289152/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827579.074061:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827579.074069:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827579.074075:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790288512:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827579.074076:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827579.074084:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787827579.074085:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790288640:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827579.074088:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827579.074095:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4215 00000100:00080000:3.0:1787827579.074100:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827579.074105:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787827579.074110:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787827579.074110:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827579.074123:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790289280/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827579.074130:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827579.074133:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827579/real 1787827579] req@ffff9e680a2d4780 x1874668790288640/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827579.074134:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827579.074137:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827579.074145:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4215 00000100:00080000:3.0:1787827579.074151:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827579.074154:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790288640/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827579.074156:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827579.074163:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787827579.074170:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827579.074175:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827579.074175:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790289408/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827579.074182:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790288640:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827579.074193:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827579.074196:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827579.074197:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827579.074201:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4215 00000100:00080000:3.0:1787827579.074203:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827579.074206:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827579.074209:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827579.074210:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790288768:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827579.074219:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790289536/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827579.074230:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827579.074240:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827579/real 1787827579] req@ffff9e680a2d5e00 x1874668790288768/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827579.074249:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790288768/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827579.074254:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827579.074256:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827579.074258:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790288768:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827579.074262:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790288896:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827579.074272:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827579.074278:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827579/real 1787827579] req@ffff9e680a2d61c0 x1874668790288896/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827579.074286:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790288896/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827579.074291:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827579.074292:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827579.074294:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790288896:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827579.074297:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790289024:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827579.074307:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827579.074313:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827579/real 1787827579] req@ffff9e680a2d4000 x1874668790289024/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827579.074320:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790289024/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827579.074328:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827579.074331:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827579.074335:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790289024:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827579.074341:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790289152:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827579.074360:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827579.074371:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827579/real 1787827579] req@ffff9e680a2d5a40 x1874668790289152/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827579.074382:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790289152/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827579.074389:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827579.074391:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827579.074393:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790289152:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827579.074406:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790289280:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827579.074420:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827579.074426:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827579/real 1787827579] req@ffff9e680a2d4b40 x1874668790289280/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827579.074434:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790289280/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827579.074439:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827579.074440:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827579.074442:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790289280:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827579.074446:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790289408:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827579.074455:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827579.074461:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827579/real 1787827579] req@ffff9e680a2d7480 x1874668790289408/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827579.074468:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790289408/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827579.074473:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827579.074474:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827579.074477:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790289408:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827579.074491:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790289536:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827579.074502:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827579.074507:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827579/real 1787827579] req@ffff9e680a2d52c0 x1874668790289536/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827579.074514:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790289536/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827579.074520:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827579.074523:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827579.074527:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790289536:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827579.585817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827579.777856:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827580.609774:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827580.801855:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827581.633821:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827581.825919:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827582.657839:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827582.849868:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827583.681821:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827583.873815:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827584.193790:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827584.193801:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827584.193805:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827584.193823:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4220 00000100:00080000:3.0:1787827584.193835:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827584.193844:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827584.193880:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790289664/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827584.193909:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827584.193914:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827584.193917:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827584.193922:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4220 00000100:00080000:3.0:1787827584.193929:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827584.193934:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827584.193945:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827584.193950:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790289664:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787827584.193962:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790289792/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827584.193984:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827584.193989:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827584.193991:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827584.193996:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4220 00000800:00000100:1.0:1787827584.194021:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827584.194057:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827584/real 1787827584] req@ffff9e680a2d6d00 x1874668790289664/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827584.194076:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827584.194081:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790289664/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827584.194087:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787827584.194092:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787827584.194097:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787827584.194098:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827584.194105:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790289664:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787827584.194118:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790289920/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827584.194137:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790289792:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827584.194156:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827584.194167:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827584/real 1787827584] req@ffff9e680a2d6580 x1874668790289792/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827584.194175:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790289792/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827584.194182:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827584.194184:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827584.194186:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790289792:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827584.194193:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790289920:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827584.194205:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827584.194228:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827584/real 1787827584] req@ffff9e680a2d43c0 x1874668790289920/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827584.194240:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790289920/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827584.194254:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827584.194257:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827584.194260:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790289920:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827584.194274:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827584.194296:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827584.194299:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827584.194304:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4220 00000100:00080000:3.0:1787827584.194309:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827584.194315:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827584.194369:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827584.194386:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790290048/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827584.194406:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827584.194422:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827584.194425:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827584.194430:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790290048:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787827584.194431:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4220 00000100:00080000:3.0:1787827584.194440:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827584.194445:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827584.194450:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787827584.194455:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827584.194466:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827584/real 1787827584] req@ffff9e680a2d70c0 x1874668790290048/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827584.194474:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790290176/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827584.194479:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790290048/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827584.194517:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827584.194519:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827584.194522:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827584.194528:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787827584.194529:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790290048:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787827584.194534:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827584.194542:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4220 00000100:00100000:1.0:1787827584.194545:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790290176:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787827584.194549:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827584.194557:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827584.194563:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827584.194567:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827584.194578:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827584/real 1787827584] req@ffff9e680a2d7c00 x1874668790290176/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827584.194583:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790290304/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827584.194593:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790290176/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827584.194606:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827584.194607:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827584.194614:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827584.194615:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787827584.194620:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790290176:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787827584.194620:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827584.194630:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4220 00000100:00080000:3.0:1787827584.194635:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827584.194636:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790290304:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827584.194644:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827584.194651:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827584.194656:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827584.194666:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827584/real 1787827584] req@ffff9e680a2d70c0 x1874668790290304/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827584.194671:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790290432/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827584.194679:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790290304/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827584.194723:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827584.194732:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827584.194733:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827584.194740:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787827584.194741:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790290304:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827584.194745:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827584.194753:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4220 00000100:00080000:3.0:1787827584.194759:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827584.194760:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790290432:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827584.194768:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827584.194775:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787827584.194785:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827584.194792:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790290560/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827584.194797:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827584/real 1787827584] req@ffff9e680a2d43c0 x1874668790290432/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827584.194822:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790290432/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827584.194824:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827584.194834:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827584.194839:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827584.194840:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827584.194848:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827584.194848:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4220 00000100:00100000:1.0:1787827584.194855:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790290432:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827584.194857:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827584.194864:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827584.194867:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790290560:192.168.203.109@tcp:8:kworker.0 00000008:00100000:3.0:1787827584.194873:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787827584.194885:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827584.194891:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790290688/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827584.194898:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827584/real 1787827584] req@ffff9e680a2d6580 x1874668790290560/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827584.194919:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790290560/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827584.194928:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827584.194932:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827584.194936:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790290560:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827584.194949:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790290688:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827584.194966:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827584.194974:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827584/real 1787827584] req@ffff9e680a2d6d00 x1874668790290688/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827584.194982:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790290688/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827584.194988:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827584.194990:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827584.194992:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790290688:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827584.705811:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827584.897887:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827585.729819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827585.922003:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827586.753764:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827586.945869:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827587.777790:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827587.969822:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827588.801766:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827588.993849:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827589.313772:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827589.313781:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827589.313784:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827589.313791:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4226 00000100:00080000:3.0:1787827589.313800:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827589.313806:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827589.313833:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790290816/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827589.313856:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827589.313859:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827589.313860:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827589.313863:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4226 00000100:00080000:3.0:1787827589.313866:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827589.313868:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827589.313875:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827589.313883:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790290944/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827589.313891:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827589.313893:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827589.313895:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827589.313897:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4226 00000100:00080000:3.0:1787827589.313899:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827589.313903:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827589.313905:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790290816:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787827589.313908:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827589.313919:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790291072/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827589.313925:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827589.313927:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827589.313928:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827589.313930:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4226 00000100:00080000:3.0:1787827589.313932:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827589.313934:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827589.313937:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827589.313944:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790291200/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827589.313949:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827589.313951:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827589.313952:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827589.313954:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4226 00000100:00080000:3.0:1787827589.313956:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827589.313958:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827589.313961:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827589.313968:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790291328/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827589.313974:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827589.313977:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827589.313978:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827589.313984:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4226 00000800:00000100:1.0:1787827589.313985:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827589.313987:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827589.313991:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827589.313995:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827589.314003:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790291456/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827589.314009:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827589.314011:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827589.314013:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827589.314016:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4226 00000100:00080000:3.0:1787827589.314019:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827589.314021:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827589.314024:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827589.314025:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827589/real 1787827589] req@ffff9e680a2d52c0 x1874668790290816/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827589.314034:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790291584/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827589.314049:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790290816/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827589.314049:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827589.314054:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827589.314058:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827589.314066:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4226 00000100:00080000:1.0:1787827589.314067:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827589.314069:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827589.314073:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827589.314076:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827589.314081:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790290816:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787827589.314081:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827589.314093:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790290944:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827589.314096:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790291712/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827589.314104:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827589.314107:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827589.314108:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827589.314111:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4226 00000100:00080000:3.0:1787827589.314114:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827589.314118:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827589.314121:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827589.314126:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827589.314132:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827589/real 1787827589] req@ffff9e680a2d7480 x1874668790290944/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827589.314136:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790291840/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827589.314157:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790290944/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827589.314165:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827589.314167:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827589.314171:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790290944:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827589.314190:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790291072:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827589.314202:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827589.314209:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827589/real 1787827589] req@ffff9e680a2d4b40 x1874668790291072/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827589.314216:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790291072/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827589.314226:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827589.314228:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827589.314230:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790291072:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827589.314234:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790291200:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827589.314243:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827589.314249:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827589/real 1787827589] req@ffff9e680a2d5a40 x1874668790291200/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827589.314255:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790291200/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827589.314260:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827589.314262:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827589.314264:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790291200:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827589.314267:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790291328:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827589.314276:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827589.314282:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827589/real 1787827589] req@ffff9e680a2d4000 x1874668790291328/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827589.314289:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790291328/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827589.314294:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827589.314296:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827589.314298:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790291328:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827589.314302:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790291456:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827589.314311:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827589.314316:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827589/real 1787827589] req@ffff9e680a2d61c0 x1874668790291456/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827589.314323:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790291456/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827589.314329:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827589.314331:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827589.314333:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790291456:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827589.314336:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790291584:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827589.314345:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827589.314351:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827589/real 1787827589] req@ffff9e680a2d5e00 x1874668790291584/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827589.314358:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790291584/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827589.314363:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827589.314364:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827589.314366:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790291584:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827589.314369:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790291712:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827589.314379:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827589.314384:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827589/real 1787827589] req@ffff9e680a2d4780 x1874668790291712/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827589.314391:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790291712/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827589.314396:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827589.314398:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827589.314400:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790291712:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827589.314403:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790291840:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827589.314412:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827589.314417:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827589/real 1787827589] req@ffff9e680a2d4f00 x1874668790291840/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827589.314424:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790291840/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827589.314429:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827589.314431:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827589.314433:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790291840:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827589.825752:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827590.017903:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827590.849848:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827591.041934:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827591.873775:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827592.065886:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827592.897885:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827593.089963:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827593.921791:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827594.113887:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827594.433860:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827594.433876:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827594.433883:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827594.433901:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4231 00000100:00080000:3.0:1787827594.433935:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827594.433947:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827594.434024:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790291968/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827594.434085:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827594.434093:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827594.434097:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827594.434106:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4231 00000100:00080000:3.0:1787827594.434114:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827594.434123:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827594.434139:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827594.434168:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790292096/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827594.434190:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790291968:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827594.434236:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827594.434245:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827594.434254:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827594.434263:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4231 00000100:00080000:3.0:1787827594.434272:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827594.434299:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827594.434311:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827594.434330:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827594.434343:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790292224/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827594.434365:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827594.434372:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827594.434377:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827594.434386:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4231 00000100:00100000:1.0:1787827594.434392:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827594/real 1787827594] req@ffff9e680a2d4f00 x1874668790291968/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827594.434397:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827594.434415:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827594.434462:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827594.434480:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790291968/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827594.434494:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790292352/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827594.434502:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827594.434521:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827594.434533:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827594.434545:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787827594.434546:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790291968:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827594.434552:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827594.434566:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4231 00000100:00080000:3.0:1787827594.434574:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827594.434583:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827594.434588:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790292096:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787827594.434599:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827594.434630:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790292480/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827594.434645:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827594.434667:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827594.434676:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827594/real 1787827594] req@ffff9e680a2d4780 x1874668790292096/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827594.434680:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827594.434688:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827594.434748:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790292096/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827594.434759:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4231 00000100:00080000:3.0:1787827594.434778:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827594.434779:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827594.434793:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827594.434796:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827594.434805:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790292096:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787827594.434814:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827594.434834:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790292224:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827594.434847:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790292608/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827594.434877:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827594.434887:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827594.434896:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827594.434901:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827594.434921:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827594/real 1787827594] req@ffff9e680a2d5e00 x1874668790292224/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827594.434932:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4231 00000100:00080000:3.0:1787827594.434946:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827594.434953:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790292224/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827594.434960:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827594.434993:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787827594.435020:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787827594.435025:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790292736/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827594.435036:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827594.435057:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790292224:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827594.435059:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827594.435074:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827594.435080:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827594.435089:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790292352:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787827594.435090:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4231 00000100:00080000:3.0:1787827594.435102:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827594.435113:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827594.435122:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827594.435145:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790292864/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827594.435154:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827594.435178:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827594.435183:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827594/real 1787827594] req@ffff9e680a2d61c0 x1874668790292352/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827594.435191:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827594.435202:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827594.435218:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4231 00000100:00100000:1.0:1787827594.435221:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790292352/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827594.435249:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827594.435258:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827594.435264:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787827594.435270:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827594.435278:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790292352:192.168.203.109@tcp:8:kworker.0 00000008:00100000:3.0:1787827594.435281:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827594.435324:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790292992/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827594.435331:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790292480:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827594.435381:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827594.435402:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827594/real 1787827594] req@ffff9e680a2d4000 x1874668790292480/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827594.435417:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790292480/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827594.435475:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827594.435480:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827594.435484:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790292480:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827594.435492:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790292608:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827594.435536:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827594.435553:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827594/real 1787827594] req@ffff9e680a2d5a40 x1874668790292608/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827594.435568:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790292608/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827594.435580:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827594.435584:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827594.435588:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790292608:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827594.435595:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790292736:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827594.435615:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827594.435627:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827594/real 1787827594] req@ffff9e680a2d4b40 x1874668790292736/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827594.435641:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790292736/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827594.435656:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827594.435663:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827594.435670:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790292736:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827594.435683:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790292864:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827594.435759:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827594.435780:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827594/real 1787827594] req@ffff9e680a2d7480 x1874668790292864/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827594.435795:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790292864/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827594.435806:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827594.435810:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827594.435814:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790292864:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827594.435837:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790292992:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827594.435864:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827594.435876:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827594/real 1787827594] req@ffff9e680a2d52c0 x1874668790292992/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827594.435890:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790292992/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827594.435900:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827594.435903:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827594.435907:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790292992:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827594.945815:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827595.137896:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827595.969858:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827596.161800:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827596.993811:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827597.185922:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827598.017837:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827598.209945:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827599.041785:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827599.233858:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827599.553783:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827599.553792:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827599.553795:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827599.553803:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4236 00000100:00080000:0.0:1787827599.553812:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827599.553818:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827599.553849:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790293120/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827599.553872:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827599.553875:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827599.553876:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827599.553879:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4236 00000100:00080000:0.0:1787827599.553881:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827599.553884:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827599.553891:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827599.553900:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790293248/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827599.553908:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827599.553912:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827599.553915:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827599.553916:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790293120:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787827599.553919:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4236 00000100:00080000:0.0:1787827599.553924:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827599.553927:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827599.553929:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827599.553941:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790293376/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827599.553951:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827599.553955:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827599.553957:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827599.553961:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4236 00000100:00080000:0.0:1787827599.553967:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827599.553972:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1787827599.553977:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787827599.553981:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827599.553995:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790293504/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827599.554002:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827599.554005:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827599.554006:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827599.554009:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4236 00000100:00080000:0.0:1787827599.554013:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827599.554014:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827599/real 1787827599] req@ffff9e6803e98b40 x1874668790293120/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827599.554018:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827599.554026:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827599.554039:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790293632/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827599.554044:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790293120/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827599.554050:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827599.554056:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827599.554059:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827599.554064:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4236 00000100:00080000:1.0:1787827599.554064:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827599.554069:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827599.554073:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787827599.554076:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827599.554081:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790293120:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1787827599.554082:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827599.554094:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790293760/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827599.554101:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827599.554105:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790293248:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827599.554106:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827599.554109:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827599.554115:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4236 00000100:00080000:0.0:1787827599.554118:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827599.554120:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827599.554124:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827599.554131:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827599.554143:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790293888/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827599.554145:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827599/real 1787827599] req@ffff9e6803e9bc00 x1874668790293248/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827599.554157:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827599.554164:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827599.554166:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827599.554172:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4236 00000100:00080000:0.0:1787827599.554174:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827599.554178:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790293248/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827599.554180:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827599.554186:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827599.554196:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790294016/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827599.554206:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827599.554209:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827599.554210:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827599.554211:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827599.554220:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787827599.554221:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4236 00000100:00080000:0.0:1787827599.554227:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827599.554230:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790293248:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827599.554232:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827599.554240:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827599.554243:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790293376:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827599.554248:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790294144/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827599.554266:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827599.554276:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827599/real 1787827599] req@ffff9e6803e992c0 x1874668790293376/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827599.554284:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790293376/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827599.554293:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827599.554296:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827599.554298:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790293376:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827599.554302:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790293504:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827599.554312:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827599.554317:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827599/real 1787827599] req@ffff9e6803e9ad00 x1874668790293504/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827599.554324:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790293504/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827599.554329:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827599.554331:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827599.554333:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790293504:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827599.554337:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790293632:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827599.554346:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827599.554352:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827599/real 1787827599] req@ffff9e6803e98f00 x1874668790293632/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827599.554359:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790293632/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827599.554364:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827599.554365:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827599.554368:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790293632:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827599.554371:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790293760:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827599.554380:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827599.554385:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827599/real 1787827599] req@ffff9e6803e99e00 x1874668790293760/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827599.554407:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790293760/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827599.554415:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827599.554417:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827599.554419:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790293760:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827599.554433:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790293888:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827599.554453:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827599.554461:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827599/real 1787827599] req@ffff9e6803e99680 x1874668790293888/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827599.554468:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790293888/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827599.554473:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827599.554475:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827599.554477:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790293888:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827599.554481:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790294016:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827599.554490:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827599.554496:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827599/real 1787827599] req@ffff9e6803e9a940 x1874668790294016/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827599.554503:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790294016/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827599.554508:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827599.554510:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827599.554511:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790294016:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827599.554514:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790294144:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827599.554523:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827599.554529:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827599/real 1787827599] req@ffff9e6803e9b840 x1874668790294144/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827599.554542:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790294144/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827599.554547:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827599.554549:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827599.554551:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790294144:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827600.065851:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827600.257821:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827601.089790:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827601.281898:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827602.113803:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827602.305877:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827603.137807:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827603.329845:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827604.161774:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827604.353826:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827604.673759:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827604.673769:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827604.673774:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827604.673783:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4241 00000100:00080000:0.0:1787827604.673794:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827604.673802:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827604.673840:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790294272/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827604.673873:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827604.673881:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827604.673882:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827604.673886:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4241 00000100:00080000:0.0:1787827604.673890:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827604.673894:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827604.673902:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827604.673910:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790294272:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827604.673915:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790294400/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827604.673924:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827604.673926:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827604.673928:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827604.673931:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4241 00000100:00080000:0.0:1787827604.673935:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827604.673939:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827604.673944:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827604.673956:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790294528/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827604.673967:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787827604.673970:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827604.673974:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827604.673978:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827604.673984:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4241 00000100:00080000:0.0:1787827604.673990:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827604.673994:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827604.673999:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827604.674008:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827604/real 1787827604] req@ffff9e6803e9a1c0 x1874668790294272/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827604.674011:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790294656/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827604.674032:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827604.674035:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827604.674036:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827604.674039:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4241 00000100:00080000:0.0:1787827604.674044:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827604.674050:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827604.674055:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827604.674062:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790294272/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827604.674066:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790294784/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827604.674087:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827604.674089:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827604.674095:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827604.674097:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787827604.674099:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827604.674105:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790294272:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787827604.674108:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4241 00000100:00080000:0.0:1787827604.674116:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827604.674121:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827604.674125:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827604.674138:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790294912/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827604.674138:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790294400:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827604.674157:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827604.674160:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827604.674162:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827604.674165:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4241 00000800:00000100:1.0:1787827604.674169:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827604.674170:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827604.674176:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827604.674180:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827604.674186:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827604/real 1787827604] req@ffff9e6803e9a580 x1874668790294400/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827604.674191:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790295040/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827604.674211:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827604.674213:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827604.674217:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827604.674224:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4241 00000100:00080000:0.0:1787827604.674226:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827604.674230:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790294400/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827604.674233:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827604.674239:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787827604.674246:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787827604.674251:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790295168/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827604.674254:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827604.674261:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790294400:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827604.674272:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827604.674273:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790294528:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827604.674277:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827604.674280:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827604.674283:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4241 00000100:00080000:0.0:1787827604.674287:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827604.674290:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827604.674293:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787827604.674297:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827604.674307:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790295296/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827604.674310:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827604/real 1787827604] req@ffff9e6803e98780 x1874668790294528/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827604.674340:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790294528/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827604.674350:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827604.674353:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827604.674370:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790294528:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827604.674375:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790294656:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827604.674390:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827604.674398:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827604/real 1787827604] req@ffff9e6803e9b480 x1874668790294656/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827604.674406:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790294656/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827604.674422:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827604.674424:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827604.674426:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790294656:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827604.674430:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790294784:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827604.674440:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827604.674446:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827604/real 1787827604] req@ffff9e6803e9b840 x1874668790294784/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827604.674453:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790294784/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827604.674458:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827604.674460:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827604.674462:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790294784:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827604.674473:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790294912:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827604.674483:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827604.674489:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827604/real 1787827604] req@ffff9e6803e9a940 x1874668790294912/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827604.674496:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790294912/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827604.674502:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827604.674504:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827604.674505:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790294912:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827604.674509:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790295040:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827604.674518:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827604.674532:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827604/real 1787827604] req@ffff9e6803e99680 x1874668790295040/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827604.674539:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790295040/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827604.674544:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827604.674545:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827604.674547:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790295040:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827604.674551:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790295168:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827604.674560:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827604.674566:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827604/real 1787827604] req@ffff9e6803e99e00 x1874668790295168/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827604.674573:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790295168/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827604.674578:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827604.674580:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827604.674582:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790295168:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827604.674585:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790295296:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827604.674594:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827604.674599:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827604/real 1787827604] req@ffff9e6803e98f00 x1874668790295296/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827604.674606:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790295296/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827604.674611:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827604.674613:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827604.674615:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790295296:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827605.185788:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827605.377846:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827606.209857:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827606.401853:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827607.233789:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827607.425845:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827608.257789:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827608.449826:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827609.281819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827609.473965:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827609.793903:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827609.793912:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827609.793917:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827609.793938:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4246 00000100:00080000:1.0:1787827609.793950:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827609.794007:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827609.794048:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790295424/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827609.794082:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827609.794086:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827609.794088:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827609.794092:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4246 00000100:00080000:1.0:1787827609.794095:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827609.794099:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827609.794111:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827609.794129:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790295552/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827609.794144:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827609.794149:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827609.794153:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827609.794159:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4246 00000100:00080000:1.0:1787827609.794166:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827609.794173:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827609.794181:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827609.794208:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790295680/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827609.794210:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790295424:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827609.794234:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827609.794238:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827609.794241:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827609.794246:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4246 00000100:00080000:1.0:1787827609.794251:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827609.794256:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827609.794262:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827609.794275:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790295808/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827609.794285:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827609.794291:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1787827609.794292:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827609.794294:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827609.794302:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4246 00000100:00080000:1.0:1787827609.794307:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827609.794311:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827609.794317:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827609.794359:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790295936/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827609.794400:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827609/real 1787827609] req@ffff9e6831aaad00 x1874668790295424/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827609.794417:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827609.794428:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827609.794432:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827609.794437:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790295424/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827609.794449:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4246 00000100:00080000:2.0:1787827609.794462:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827609.794465:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827609.794473:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787827609.794487:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827609.794488:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790295424:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787827609.794517:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827609.794519:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790295552:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827609.794580:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790296064/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827609.794595:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827609.794600:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827609.794604:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787827609.794612:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827609.794620:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4246 00000100:00080000:1.0:1787827609.794628:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827609.794635:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827609.794649:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827609.794659:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827609/real 1787827609] req@ffff9e6831aa8b40 x1874668790295552/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827609.794667:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790296192/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827609.794714:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827609.794718:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827609.794720:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827609.794729:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4246 00000100:00100000:2.0:1787827609.794729:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790295552/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827609.794787:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827609.794794:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827609.794806:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787827609.794808:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827609.794829:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827609.794836:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790296320/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827609.794841:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790295552:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827609.794866:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827609.794870:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787827609.794900:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790295680:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827609.794910:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827609.794919:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4246 00000100:00080000:1.0:1787827609.794924:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827609.794929:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827609.794936:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827609.794947:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827609.794957:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790296448/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827609.794970:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827609/real 1787827609] req@ffff9e6831aa8000 x1874668790295680/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827609.794991:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790295680/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827609.795001:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827609.795006:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827609.795011:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790295680:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827609.795019:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790295808:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827609.795039:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827609.795050:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827609/real 1787827609] req@ffff9e6831aab480 x1874668790295808/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827609.795063:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790295808/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827609.795072:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827609.795075:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827609.795078:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790295808:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827609.795084:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790295936:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827609.795101:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827609.795111:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827609/real 1787827609] req@ffff9e6831aa9a40 x1874668790295936/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827609.795123:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790295936/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827609.795133:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827609.795137:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827609.795140:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790295936:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827609.795146:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790296064:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827609.795163:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827609.795173:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827609/real 1787827609] req@ffff9e6831aaa1c0 x1874668790296064/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827609.795184:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790296064/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827609.795195:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827609.795198:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827609.795201:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790296064:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827609.795211:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790296192:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827609.795246:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827609.795258:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827609/real 1787827609] req@ffff9e6831aa8780 x1874668790296192/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827609.795272:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790296192/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827609.795281:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827609.795285:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827609.795289:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790296192:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827609.795296:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790296320:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827609.795316:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827609.795355:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827609/real 1787827609] req@ffff9e6831aa8f00 x1874668790296320/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827609.795370:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790296320/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827609.795389:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827609.795393:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827609.795397:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790296320:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827609.795421:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790296448:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827609.795447:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827609.795459:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827609/real 1787827609] req@ffff9e6831aaa940 x1874668790296448/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827609.795473:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790296448/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827609.795483:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827609.795498:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827609.795501:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790296448:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827610.305822:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827610.497886:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827611.329819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827611.522086:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827612.353812:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827612.545925:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827613.377843:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827613.569937:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827614.401844:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827614.593932:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827614.913804:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827614.913813:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827614.913818:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827614.913827:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4251 00000100:00080000:0.0:1787827614.913836:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827614.913842:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827614.913877:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790296576/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827614.913903:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827614.913906:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827614.913908:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827614.913911:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4251 00000100:00080000:0.0:1787827614.913914:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827614.913917:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827614.913925:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827614.913934:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790296704/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827614.913943:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827614.913949:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827614.913950:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827614.913953:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4251 00000100:00080000:0.0:1787827614.913956:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827614.913959:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827614.913963:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827614.913969:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790296576:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827614.913974:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790296832/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827614.913986:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827614.913988:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827614.913990:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827614.913993:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4251 00000100:00080000:0.0:1787827614.913996:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827614.913998:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827614.914002:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827614.914013:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790296960/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827614.914024:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827614.914028:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827614.914031:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827614.914035:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4251 00000100:00080000:0.0:1787827614.914041:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827614.914047:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827614.914063:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827614.914071:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827614.914088:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790297088/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827614.914096:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827614.914099:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827614.914101:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827614.914107:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4251 00000100:00100000:2.0:1787827614.914110:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827614/real 1787827614] req@ffff9e6803e9ad00 x1874668790296576/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827614.914111:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827614.914120:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827614.914128:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827614.914145:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790296576/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827614.914156:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790297216/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827614.914161:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827614.914166:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827614.914171:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827614.914172:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787827614.914173:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827614.914184:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4251 00000100:00100000:2.0:1787827614.914184:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790296576:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787827614.914187:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827614.914194:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827614.914200:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827614.914203:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790296704:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827614.914211:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790297344/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827614.914221:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827614.914223:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827614.914226:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827614.914230:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4251 00000100:00080000:0.0:1787827614.914235:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827614.914240:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787827614.914243:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787827614.914248:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827614.914260:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790297472/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827614.914267:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827614/real 1787827614] req@ffff9e6803e992c0 x1874668790296704/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827614.914272:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827614.914280:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827614.914303:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827614.914312:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4251 00000100:00080000:0.0:1787827614.914316:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827614.914321:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790296704/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827614.914323:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827614.914341:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787827614.914354:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787827614.914356:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790297600/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827614.914366:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827614.914376:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790296704:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827614.914406:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790296832:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827614.914432:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827614.914447:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827614/real 1787827614] req@ffff9e6803e9bc00 x1874668790296832/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827614.914463:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790296832/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827614.914474:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827614.914478:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827614.914483:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790296832:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827614.914490:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790296960:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827614.914523:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827614.914537:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827614/real 1787827614] req@ffff9e6803e98b40 x1874668790296960/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827614.914552:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790296960/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827614.914563:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827614.914568:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827614.914572:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790296960:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827614.914580:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790297088:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827614.914601:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827614.914614:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827614/real 1787827614] req@ffff9e6803e98f00 x1874668790297088/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827614.914639:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790297088/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827614.914650:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827614.914654:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827614.914658:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790297088:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827614.914665:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790297216:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827614.914685:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827614.914723:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827614/real 1787827614] req@ffff9e6803e99e00 x1874668790297216/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827614.914741:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790297216/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827614.914753:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827614.914757:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827614.914761:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790297216:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827614.914769:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790297344:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827614.914802:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827614.914817:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827614/real 1787827614] req@ffff9e6803e99680 x1874668790297344/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827614.914832:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790297344/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827614.914851:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827614.914855:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827614.914860:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790297344:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827614.914868:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790297472:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827614.914890:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827614.914902:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827614/real 1787827614] req@ffff9e6803e9a940 x1874668790297472/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827614.914917:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790297472/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827614.914927:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827614.914931:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827614.914935:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790297472:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827614.914943:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790297600:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827614.914964:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827614.914977:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827614/real 1787827614] req@ffff9e6803e9b840 x1874668790297600/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827614.914993:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790297600/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827614.915003:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827614.915007:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827614.915012:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790297600:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827615.425813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827615.617970:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827616.449878:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827616.641844:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827617.473811:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827617.665926:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827618.497790:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827618.689887:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827619.521799:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827619.714011:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827620.033814:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827620.033822:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827620.033826:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827620.033836:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4256 00000100:00080000:3.0:1787827620.033848:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827620.033855:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827620.033891:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790297728/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827620.033925:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827620.033931:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827620.033934:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827620.033940:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4256 00000100:00080000:3.0:1787827620.033945:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827620.033951:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827620.033962:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790297728:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787827620.033962:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827620.033987:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790297856/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827620.034026:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827620.034031:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827620.034034:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787827620.034039:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827620.034044:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4256 00000100:00080000:3.0:1787827620.034050:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827620.034055:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827620.034061:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827620.034075:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790297984/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827620.034083:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827620/real 1787827620] req@ffff9e680a2d6d00 x1874668790297728/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827620.034087:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827620.034098:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827620.034103:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827620.034110:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790297728/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827620.034111:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4256 00000100:00080000:3.0:1787827620.034120:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827620.034125:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827620.034131:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827620.034135:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790297728:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827620.034155:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790297856:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827620.034174:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827620.034183:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827620/real 1787827620] req@ffff9e680a2d6580 x1874668790297856/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827620.034194:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790297856/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827620.034210:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827620.034214:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827620.034217:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827620.034221:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790297856:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787827620.034226:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827620.034230:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790297984:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827620.034247:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790298112/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827620.034281:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827620.034296:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827620.034301:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787827620.034302:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827620/real 1787827620] req@ffff9e680a2d43c0 x1874668790297984/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827620.034304:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827620.034314:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4256 00000100:00080000:3.0:1787827620.034321:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827620.034326:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790297984/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827620.034329:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827620.034338:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787827620.034340:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827620.034345:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827620.034349:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790297984:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827620.034354:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790298240/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827620.034365:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827620.034368:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790298112:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787827620.034369:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827620.034375:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827620.034380:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4256 00000100:00080000:3.0:1787827620.034384:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827620.034388:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827620.034393:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787827620.034396:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827620.034408:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827620/real 1787827620] req@ffff9e680a2d70c0 x1874668790298112/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827620.034414:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790298368/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827620.034424:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790298112/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827620.034439:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827620.034440:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827620.034447:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827620.034450:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787827620.034452:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790298112:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787827620.034458:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827620.034466:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790298240:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787827620.034468:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4256 00000100:00080000:3.0:1787827620.034478:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827620.034485:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787827620.034488:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827620.034523:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827620/real 1787827620] req@ffff9e680a2d7c00 x1874668790298240/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827620.034535:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790298240/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827620.034544:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827620.034547:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827620.034550:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790298240:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827620.034558:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790298368:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827620.034602:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827620.034620:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827620/real 1787827620] req@ffff9e680a2d43c0 x1874668790298368/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827620.034677:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790298368/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827620.034706:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827620.034714:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827620.034723:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790298368:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787827620.034746:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827620.034851:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790298496/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827620.034886:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827620.034894:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827620.034899:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827620.034910:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4256 00000100:00080000:3.0:1787827620.034920:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827620.034931:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827620.034960:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790298496:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787827620.034962:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827620.034986:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790298624/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827620.034998:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827620.035001:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827620.035008:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827620.035028:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827620.035037:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4256 00000100:00100000:0.0:1787827620.035044:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827620/real 1787827620] req@ffff9e680a2d6580 x1874668790298496/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827620.035048:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827620.035065:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827620.035082:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827620.035098:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790298496/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827620.035111:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790298752/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827620.035127:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827620.035143:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827620.035154:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790298496:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827620.035185:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790298624:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827620.035220:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827620.035235:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827620/real 1787827620] req@ffff9e680a2d6d00 x1874668790298624/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827620.035281:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790298624/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827620.035318:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827620.035325:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827620.035332:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790298624:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827620.035343:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790298752:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827620.035379:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827620.035400:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827620/real 1787827620] req@ffff9e680a2d52c0 x1874668790298752/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827620.035415:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790298752/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827620.035425:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827620.035428:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827620.035432:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790298752:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827620.545795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827620.737856:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827621.569821:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827621.761891:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827622.593777:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827622.785921:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827623.617813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827623.809905:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827624.641798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827624.833937:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827625.153868:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827625.153880:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827625.153886:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827625.153897:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4261 00000100:00080000:3.0:1787827625.153910:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827625.153920:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827625.153981:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790298880/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827625.154026:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827625.154030:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827625.154033:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827625.154037:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4261 00000100:00080000:3.0:1787827625.154041:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827625.154045:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827625.154058:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827625.154083:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790299008/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827625.154098:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827625.154104:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790298880:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827625.154104:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827625.154110:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827625.154125:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4261 00000100:00080000:3.0:1787827625.154128:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827625.154133:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827625.154138:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827625.154151:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790299136/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827625.154179:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827625.154183:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827625.154186:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827625.154190:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4261 00000100:00080000:3.0:1787827625.154194:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827625.154199:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787827625.154206:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787827625.154210:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827625.154266:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790299264/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827625.154278:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827625.154282:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827625.154284:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827625.154289:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4261 00000100:00080000:3.0:1787827625.154295:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827625.154305:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827625/real 1787827625] req@ffff9e680a2d7480 x1874668790298880/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827625.154306:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827625.154326:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827625.154346:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790299392/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827625.154357:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790298880/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827625.154367:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827625.154380:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827625.154385:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827625.154392:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827625.154394:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4261 00000100:00080000:3.0:1787827625.154404:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827625.154407:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827625.154415:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827625.154419:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790298880:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787827625.154423:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827625.154442:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790299520/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827625.154458:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827625.154463:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790299008:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827625.154465:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827625.154472:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827625.154480:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4261 00000100:00080000:3.0:1787827625.154484:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827625.154490:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827625.154496:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787827625.154509:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827625.154516:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790299648/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827625.154531:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827625.154537:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827625/real 1787827625] req@ffff9e680a2d4b40 x1874668790299008/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827625.154538:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827625.154544:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827625.154556:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4261 00000100:00080000:3.0:1787827625.154560:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827625.154563:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827625.154568:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827625.154588:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790299776/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827625.154594:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790299008/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827625.154640:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827625.154647:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827625.154649:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827625.154655:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790299008:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827625.154672:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827625.154677:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827625.154680:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790299136:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827625.154686:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4261 00000100:00080000:3.0:1787827625.154694:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787827625.154746:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827625.154751:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827625.154765:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827625.154767:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827625/real 1787827625] req@ffff9e680a2d5a40 x1874668790299136/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827625.154789:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790299136/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827625.154801:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790299904/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827625.154805:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827625.154818:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827625.154830:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790299136:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827625.154847:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790299264:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827625.154871:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827625.154884:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827625/real 1787827625] req@ffff9e680a2d4000 x1874668790299264/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827625.154898:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790299264/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827625.154907:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827625.154910:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827625.154914:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790299264:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827625.154920:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790299392:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827625.154950:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827625.154972:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827625/real 1787827625] req@ffff9e680a2d61c0 x1874668790299392/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827625.155004:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790299392/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827625.155021:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827625.155027:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827625.155034:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790299392:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827625.155067:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790299520:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827625.155107:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827625.155121:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827625/real 1787827625] req@ffff9e680a2d5e00 x1874668790299520/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827625.155135:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790299520/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827625.155154:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827625.155157:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827625.155161:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790299520:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827625.155168:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790299648:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827625.155185:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827625.155195:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827625/real 1787827625] req@ffff9e680a2d4780 x1874668790299648/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827625.155207:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790299648/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827625.155243:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827625.155247:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827625.155251:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790299648:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827625.155258:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790299776:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827625.155289:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827625.155301:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827625/real 1787827625] req@ffff9e680a2d4f00 x1874668790299776/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827625.155315:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790299776/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827625.155324:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827625.155327:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827625.155332:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790299776:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827625.155338:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790299904:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827625.155355:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827625.155365:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827625/real 1787827625] req@ffff9e680a2d7480 x1874668790299904/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827625.155377:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790299904/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827625.155386:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827625.155389:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827625.155393:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790299904:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827625.665749:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827625.857939:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827626.689764:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827626.881898:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827627.713763:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827627.905909:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827628.737769:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827628.929816:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827629.761777:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827629.953842:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827630.273772:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827630.273782:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827630.273785:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827630.273794:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4266 00000100:00080000:3.0:1787827630.273804:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827630.273810:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827630.273846:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790300032/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827630.273876:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827630.273879:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827630.273881:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827630.273884:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4266 00000100:00080000:3.0:1787827630.273888:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827630.273893:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827630.273902:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827630.273912:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790300032:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787827630.273923:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790300160/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827630.273937:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827630.273940:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827630.273941:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827630.273944:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4266 00000100:00080000:3.0:1787827630.273948:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827630.273950:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827630.273953:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827630.273962:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790300288/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787827630.273970:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827630.273974:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827630.273979:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827630.273982:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827630.273987:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4266 00000100:00080000:3.0:1787827630.273992:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827630.274014:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827630.274022:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827630.274038:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827630/real 1787827630] req@ffff9e680a2d52c0 x1874668790300032/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827630.274057:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790300416/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827630.274066:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790300032/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827630.274091:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827630.274100:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827630.274108:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827630.274110:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827630.274112:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827630.274118:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790300032:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827630.274123:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4266 00000100:00080000:3.0:1787827630.274130:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827630.274134:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827630.274139:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827630.274149:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790300160:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827630.274155:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790300544/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827630.274168:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827630.274172:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827630.274173:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827630.274195:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4266 00000100:00080000:3.0:1787827630.274200:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787827630.274206:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827630.274209:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827630.274217:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827630.274243:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827630/real 1787827630] req@ffff9e680a2d6d00 x1874668790300160/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827630.274247:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790300672/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827630.274264:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787827630.274267:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790300160/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827630.274269:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827630.274273:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827630.274282:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4266 00000100:00080000:0.0:1787827630.274285:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827630.274286:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827630.274291:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827630.274294:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827630.274298:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790300160:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787827630.274301:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827630.274305:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790300288:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827630.274311:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790300800/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827630.274324:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787827630.274325:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827630.274330:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827630.274333:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827630.274336:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827630/real 1787827630] req@ffff9e680a2d6580 x1874668790300288/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827630.274337:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4266 00000100:00080000:3.0:1787827630.274343:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827630.274350:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827630.274354:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790300288/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787827630.274354:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787827630.274364:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787827630.274368:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790300928/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827630.274370:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827630.274377:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790300288:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827630.274388:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790300416:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787827630.274388:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827630.274396:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827630.274397:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827630.274402:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4266 00000100:00080000:3.0:1787827630.274405:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827630.274408:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787827630.274412:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787827630.274416:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827630.274423:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827630/real 1787827630] req@ffff9e680a2d43c0 x1874668790300416/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827630.274428:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790301056/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827630.274441:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790300416/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827630.274449:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827630.274451:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827630.274454:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790300416:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827630.274468:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790300544:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827630.274480:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827630.274486:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827630/real 1787827630] req@ffff9e680a2d7c00 x1874668790300544/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827630.274493:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790300544/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827630.274498:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827630.274500:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827630.274502:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790300544:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827630.274506:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790300672:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827630.274515:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827630.274520:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827630/real 1787827630] req@ffff9e680a2d70c0 x1874668790300672/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827630.274527:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790300672/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827630.274533:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827630.274535:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827630.274537:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790300672:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827630.274540:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790300800:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827630.274549:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827630.274554:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827630/real 1787827630] req@ffff9e680a2d52c0 x1874668790300800/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827630.274561:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790300800/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827630.274565:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827630.274567:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827630.274569:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790300800:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827630.274573:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790300928:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827630.274581:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827630.274587:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827630/real 1787827630] req@ffff9e680a2d7480 x1874668790300928/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827630.274593:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790300928/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827630.274598:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827630.274600:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827630.274602:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790300928:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827630.274606:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790301056:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827630.274615:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827630.274621:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827630/real 1787827630] req@ffff9e680a2d4f00 x1874668790301056/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827630.274628:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790301056/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827630.274632:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827630.274634:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827630.274636:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790301056:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827630.785794:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827630.978050:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827631.809837:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827632.001847:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827632.833782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827633.025838:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827633.857778:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827634.049873:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827634.881782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827635.073945:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827635.393774:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827635.393783:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827635.393787:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827635.393797:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4272 00000100:00080000:0.0:1787827635.393807:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827635.393813:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827635.393848:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790301184/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827635.393884:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827635.393888:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827635.393890:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827635.393894:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4272 00000100:00080000:0.0:1787827635.393897:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827635.393901:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827635.393911:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827635.393922:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790301312/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827635.393932:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827635.393936:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827635.393937:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827635.393942:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4272 00000100:00080000:0.0:1787827635.393945:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827635.393951:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827635.393952:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790301184:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1787827635.393956:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827635.393970:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790301440/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827635.393981:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827635.393985:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827635.393986:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827635.393990:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4272 00000100:00080000:0.0:1787827635.394003:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827635.394008:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827635.394014:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827635.394027:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790301568/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827635.394037:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827635.394040:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827635.394042:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827635.394046:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4272 00000100:00080000:0.0:1787827635.394050:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827635.394054:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827635.394057:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790301312:192.168.203.109@tcp:38:kworker.0 00000008:00100000:0.0:1787827635.394076:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827635.394091:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790301696/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827635.394102:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827635.394104:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790301440:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827635.394109:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827635.394114:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827635.394119:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4272 00000100:00080000:0.0:1787827635.394131:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827635.394136:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827635.394138:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790301568:192.168.203.109@tcp:8:kworker.0 00000008:00100000:0.0:1787827635.394157:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827635.394174:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790301824/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827635.394182:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790301696:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787827635.394198:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827635.394203:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827635.394206:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827635.394211:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4272 00000100:00080000:0.0:1787827635.394216:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827635.394217:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790301824:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827635.394224:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827635.394231:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827635.394245:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790301952/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827635.394264:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827635.394268:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827635.394270:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827635.394275:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4272 00000100:00080000:0.0:1787827635.394280:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827635.394285:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827635.394290:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790301952:192.168.203.109@tcp:38:kworker.0 00000008:00100000:0.0:1787827635.394293:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827635.394311:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790302080/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827635.394322:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827635.394326:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790302080:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787827635.394328:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827635.394333:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827635.394339:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4272 00000100:00080000:0.0:1787827635.394345:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827635.394349:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827635.394355:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827635.394367:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790302208/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827635.394411:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790302208:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827635.905795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827636.097854:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827636.929757:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827637.121815:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827637.953792:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827638.145865:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787827638.465985:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:1.0:1787827638.466029:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787827638.466045:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787827638.466148:0:2386:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:1.0:1787827638.466172:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787827638.466188:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787827638.466199:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787827638.466214:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787827638.466225:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787827638.466238:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787827638.466241:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827635/real 1787827638] req@ffff9e6803e98f00 x1874668790302208/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787827638.466253:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787827638.466301:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790302208/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827638.466324:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827638.466334:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827638.466342:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790302208:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827638.466356:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827635/real 1787827638] req@ffff9e6803e99e00 x1874668790302080/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827638.466393:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790302080/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827638.466409:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827638.466415:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827638.466421:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790302080:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827638.466449:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827635/real 1787827638] req@ffff9e6803e99680 x1874668790301952/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827638.466469:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790301952/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827638.466492:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827638.466499:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827638.466506:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790301952:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827638.466518:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827635/real 1787827638] req@ffff9e6803e9a940 x1874668790301824/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827638.466539:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790301824/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827638.466554:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827638.466560:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827638.466567:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790301824:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827638.466579:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827635/real 1787827638] req@ffff9e6803e9b840 x1874668790301696/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827638.466601:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790301696/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827638.466620:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827638.466626:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827638.466633:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790301696:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827638.466647:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827635/real 1787827638] req@ffff9e6803e9a580 x1874668790301440/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827638.466674:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790301440/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827638.466689:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827638.466744:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827638.466753:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790301440:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827638.466765:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827635/real 1787827638] req@ffff9e6803e9a1c0 x1874668790301568/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827638.466790:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790301568/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827638.466806:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827638.466813:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827638.466820:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790301568:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827638.466832:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827635/real 1787827638] req@ffff9e6803e9b480 x1874668790301184/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827638.466860:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790301184/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827638.466877:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827638.466885:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827638.466894:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790301184:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787827638.466911:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827635/real 1787827638] req@ffff9e6803e98780 x1874668790301312/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827638.466936:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790301312/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827638.466953:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827638.466959:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827638.466966:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790301312:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827638.977796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827639.169874:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827640.001796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827640.193898:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827640.513768:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827640.513779:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827640.513783:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827640.513795:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4277 00000100:00080000:0.0:1787827640.513807:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827640.513815:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827640.513854:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790302336/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827640.513881:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827640.513884:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827640.513885:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827640.513888:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4277 00000100:00080000:0.0:1787827640.513891:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827640.513893:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827640.513902:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827640.513912:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790302464/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827640.513920:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827640.513923:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827640.513925:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827640.513927:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4277 00000100:00080000:0.0:1787827640.513929:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827640.513932:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827640.513934:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827640.513940:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790302592/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827640.513948:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827640.513951:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827640.513953:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827640.513957:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4277 00000100:00080000:0.0:1787827640.513964:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827640.513967:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790302336:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787827640.513969:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827640.513984:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827640.514013:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790302720/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827640.514025:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827640.514029:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827640.514031:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827640.514035:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4277 00000100:00080000:0.0:1787827640.514042:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827640.514045:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827640.514052:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827640.514061:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827640.514080:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790302848/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827640.514092:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827640.514099:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827640/real 1787827640] req@ffff9e6803e98b40 x1874668790302336/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827640.514100:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827640.514104:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827640.514132:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4277 00000100:00080000:0.0:1787827640.514141:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827640.514153:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827640.514160:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790302336/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787827640.514161:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787827640.514184:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787827640.514186:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790302976/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827640.514200:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827640.514212:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790302336:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787827640.514216:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827640.514229:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827640.514234:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827640.514242:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790302464:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827640.514247:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4277 00000100:00080000:0.0:1787827640.514255:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827640.514260:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827640.514266:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827640.514284:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790303104/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827640.514293:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827640.514312:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827640.514316:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827640.514318:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827640.514323:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4277 00000100:00100000:2.0:1787827640.514330:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827640/real 1787827640] req@ffff9e6803e9bc00 x1874668790302464/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827640.514331:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827640.514343:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827640.514358:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827640.514368:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790302464/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827640.514380:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790303232/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827640.514384:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827640.514397:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787827640.514409:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827640.514410:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790302464:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827640.514418:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827640.514425:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827640.514434:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4277 00000100:00080000:0.0:1787827640.514440:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827640.514445:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827640.514453:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827640.514461:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790302592:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827640.514471:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790303360/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827640.514493:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827640.514514:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827640/real 1787827640] req@ffff9e6803e992c0 x1874668790302592/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827640.514527:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790302592/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827640.514535:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827640.514538:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827640.514541:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790302592:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827640.514548:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790302720:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827640.514577:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827640.514587:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827640/real 1787827640] req@ffff9e6803e9ad00 x1874668790302720/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827640.514599:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790302720/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827640.514606:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827640.514609:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827640.514612:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790302720:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827640.514617:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790302848:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827640.514634:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827640.514644:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827640/real 1787827640] req@ffff9e6803e98780 x1874668790302848/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827640.514660:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790302848/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827640.514667:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827640.514670:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827640.514673:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790302848:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827640.514679:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790302976:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827640.514739:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827640.514753:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827640/real 1787827640] req@ffff9e6803e9b480 x1874668790302976/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827640.514766:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790302976/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827640.514782:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827640.514786:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827640.514789:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790302976:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827640.514796:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790303104:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827640.514818:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827640.514830:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827640/real 1787827640] req@ffff9e6803e9a1c0 x1874668790303104/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827640.514842:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790303104/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827640.514849:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827640.514853:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827640.514855:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790303104:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827640.514862:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790303232:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827640.514882:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827640.514894:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827640/real 1787827640] req@ffff9e6803e9a580 x1874668790303232/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827640.514923:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790303232/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827640.514932:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827640.514936:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827640.514940:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790303232:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827640.514962:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790303360:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827640.514985:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827640.514997:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827640/real 1787827640] req@ffff9e6803e9b840 x1874668790303360/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827640.515010:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790303360/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827640.515018:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827640.515022:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827640.515026:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790303360:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827641.025793:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827641.217863:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827642.049760:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827642.241879:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827643.073821:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827643.265901:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827644.097792:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827644.289963:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827645.121797:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827645.313823:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827645.633853:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827645.633867:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827645.633873:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827645.633887:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4282 00000100:00080000:1.0:1787827645.633903:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827645.633913:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827645.633964:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790303488/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827645.634001:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827645.634006:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827645.634008:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827645.634013:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4282 00000100:00080000:1.0:1787827645.634017:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827645.634021:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827645.634033:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827645.634050:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790303616/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827645.634063:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827645.634090:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827645.634093:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827645.634099:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4282 00000100:00080000:1.0:1787827645.634105:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827645.634109:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790303488:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827645.634114:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827645.634124:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827645.634144:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790303744/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827645.634158:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827645.634163:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827645.634165:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827645.634170:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4282 00000100:00080000:1.0:1787827645.634176:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827645.634181:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827645.634187:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827645.634200:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827645.634205:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790303872/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827645.634219:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827645.634224:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827645.634226:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827645.634232:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4282 00000100:00080000:1.0:1787827645.634237:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827645.634243:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827645.634250:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827645.634253:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827645/real 1787827645] req@ffff9e6831aa9680 x1874668790303488/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827645.634268:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790304000/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827645.634277:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790303488/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827645.634290:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827645.634297:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827645.634299:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827645.634300:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827645.634315:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4282 00000100:00080000:2.0:1787827645.634319:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787827645.634323:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827645.634330:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790303488:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827645.634335:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827645.634350:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827645.634356:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790303616:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827645.634369:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790304128/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827645.634381:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787827645.634389:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827645.634391:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827645.634396:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827645.634402:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4282 00000100:00080000:1.0:1787827645.634408:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827645.634409:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827645/real 1787827645] req@ffff9e6831aabc00 x1874668790303616/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827645.634418:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827645.634433:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827645.634441:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790303616/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827645.634454:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790304256/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827645.634457:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827645.634470:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827645.634480:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790303616:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827645.634495:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827645.634500:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827645.634504:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827645.634510:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790303744:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827645.634513:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4282 00000100:00080000:1.0:1787827645.634524:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827645.634531:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827645.634538:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827645.634545:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827645.634559:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790304384/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827645.634565:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827645/real 1787827645] req@ffff9e6831aa9e00 x1874668790303744/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827645.634589:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790303744/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827645.634602:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827645.634605:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827645.634609:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827645.634614:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787827645.634616:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827645.634625:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790303744:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827645.634631:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4282 00000100:00080000:1.0:1787827645.634639:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827645.634640:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790303872:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787827645.634649:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827645.634659:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827645.634672:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827645.634679:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790304512/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827645.634691:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827645/real 1787827645] req@ffff9e6831aaa580 x1874668790303872/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827645.634741:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790303872/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827645.634753:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827645.634757:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827645.634760:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790303872:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827645.634770:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790304000:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827645.634793:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827645.634803:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827645/real 1787827645] req@ffff9e6831aaa940 x1874668790304000/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827645.634813:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790304000/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827645.634821:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827645.634825:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827645.634828:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790304000:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827645.634834:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790304128:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827645.634851:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827645.634860:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827645/real 1787827645] req@ffff9e6831aa8f00 x1874668790304128/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827645.634878:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790304128/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827645.634888:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827645.634892:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827645.634894:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790304128:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827645.634900:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790304256:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827645.634920:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827645.634930:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827645/real 1787827645] req@ffff9e6831aa8780 x1874668790304256/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827645.634941:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790304256/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827645.634949:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827645.634952:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827645.634956:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790304256:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827645.634975:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790304384:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827645.634993:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827645.635003:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827645/real 1787827645] req@ffff9e6831aaa1c0 x1874668790304384/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827645.635014:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790304384/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827645.635027:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827645.635031:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827645.635034:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790304384:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827645.635040:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790304512:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827645.635059:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827645.635069:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827645/real 1787827645] req@ffff9e6831aa9a40 x1874668790304512/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827645.635097:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790304512/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827645.635104:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827645.635123:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827645.635127:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790304512:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827646.145763:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827646.337874:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827647.169813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827647.361918:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827648.193784:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827648.385899:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827649.217801:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827649.409866:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827650.241792:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827650.433957:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827650.753790:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827650.753798:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827650.753802:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827650.753810:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4287 00000100:00080000:3.0:1787827650.753819:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827650.753825:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827650.753857:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790304640/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827650.753881:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827650.753883:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827650.753885:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827650.753888:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4287 00000100:00080000:3.0:1787827650.753890:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827650.753892:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827650.753899:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827650.753907:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790304768/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827650.753914:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827650.753916:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827650.753918:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827650.753920:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4287 00000100:00080000:3.0:1787827650.753922:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827650.753926:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827650.753930:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827650.753936:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790304896/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827650.753942:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827650.753944:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827650.753945:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827650.753947:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4287 00000100:00080000:3.0:1787827650.753949:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827650.753952:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827650.753955:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827650.753961:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790305024/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827650.753968:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827650.753973:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827650.753974:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827650.753976:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790304640:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827650.753980:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4287 00000100:00080000:3.0:1787827650.753985:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827650.753993:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827650.753996:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827650.754002:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790305152/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827650.754006:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827650.754009:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827650.754010:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827650.754013:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4287 00000100:00080000:3.0:1787827650.754016:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827650.754018:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827650.754021:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827650.754027:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790305280/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827650.754033:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827650.754053:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827650.754055:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827650.754058:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4287 00000100:00080000:3.0:1787827650.754062:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827650.754065:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827650.754069:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827650.754078:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790305408/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827650.754084:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827650.754087:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827650.754089:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827650.754091:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4287 00000100:00080000:3.0:1787827650.754094:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827650.754097:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827650.754101:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827650.754108:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827650.754109:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790305536/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827650.754120:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827650.754123:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827650.754124:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827650.754127:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4287 00000100:00080000:3.0:1787827650.754129:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827650.754132:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827650.754136:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827650.754144:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790305664/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827650.754183:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827650/real 1787827650] req@ffff9e680a2d4780 x1874668790304640/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827650.754215:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790304640/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827650.754229:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827650.754234:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827650.754239:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790304640:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787827650.754249:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790304768:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827650.754276:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827650.754291:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827650/real 1787827650] req@ffff9e680a2d5e00 x1874668790304768/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827650.754305:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790304768/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827650.754315:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827650.754319:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827650.754322:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790304768:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827650.754330:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790304896:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827650.754371:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827650.754385:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827650/real 1787827650] req@ffff9e680a2d61c0 x1874668790304896/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827650.754400:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790304896/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827650.754416:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827650.754420:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827650.754424:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790304896:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827650.754460:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790305024:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827650.754485:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827650.754498:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827650/real 1787827650] req@ffff9e680a2d4000 x1874668790305024/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827650.754511:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790305024/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827650.754520:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827650.754525:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827650.754529:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790305024:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827650.754536:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790305152:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827650.754558:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827650.754570:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827650/real 1787827650] req@ffff9e680a2d5a40 x1874668790305152/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827650.754584:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790305152/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827650.754592:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827650.754596:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827650.754601:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790305152:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827650.754608:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790305280:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827650.754629:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827650.754641:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827650/real 1787827650] req@ffff9e680a2d4b40 x1874668790305280/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827650.754653:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790305280/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827650.754664:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827650.754668:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827650.754672:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790305280:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827650.754678:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790305408:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827650.754742:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827650.754755:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827650/real 1787827650] req@ffff9e680a2d4f00 x1874668790305408/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827650.754767:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790305408/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827650.754775:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827650.754778:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827650.754782:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790305408:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827650.754789:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790305536:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827650.754809:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827650.754823:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827650/real 1787827650] req@ffff9e680a2d7480 x1874668790305536/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827650.754834:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790305536/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827650.754843:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827650.754847:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827650.754850:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790305536:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827650.754857:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790305664:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827650.754878:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827650.754889:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827650/real 1787827650] req@ffff9e680a2d52c0 x1874668790305664/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827650.754902:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790305664/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827650.754909:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827650.754911:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827650.754914:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790305664:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827651.265803:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827651.457891:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827652.289765:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827652.481918:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827653.313836:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827653.505929:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827654.337832:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827654.529996:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827655.361856:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827655.553886:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827655.873825:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827655.873839:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827655.873845:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827655.873867:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4292 00000100:00080000:1.0:1787827655.873887:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827655.873899:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827655.873948:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790305792/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827655.873987:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827655.873992:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827655.873995:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827655.874017:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4292 00000100:00080000:1.0:1787827655.874023:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827655.874030:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827655.874031:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790305792:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787827655.874047:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827655.874069:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790305920/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827655.874084:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827655.874088:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827655.874091:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827655.874099:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4292 00000100:00080000:1.0:1787827655.874106:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787827655.874109:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827655.874116:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827655.874124:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827655.874140:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790306048/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827655.874152:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827655.874157:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827655.874159:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827655.874163:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4292 00000100:00080000:1.0:1787827655.874168:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827655.874173:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827655.874179:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827655.874195:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790306176/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827655.874208:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827655.874212:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827655.874216:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827655.874224:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4292 00000100:00080000:1.0:1787827655.874230:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827655.874236:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827655.874245:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827655.874261:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790306304/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827655.874272:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827655.874275:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827655.874277:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827655.874282:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4292 00000100:00080000:1.0:1787827655.874286:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827655.874292:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827655.874298:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827655.874312:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790306432/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827655.874323:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827655.874326:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827655.874328:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827655.874333:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4292 00000100:00080000:1.0:1787827655.874337:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827655.874341:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827655.874346:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827655.874360:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790306560/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827655.874371:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827655.874376:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827655/real 1787827655] req@ffff9e6831aab480 x1874668790305792/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827655.874377:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827655.874382:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827655.874394:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4292 00000100:00080000:1.0:1787827655.874400:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827655.874406:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827655.874412:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827655.874428:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790306688/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827655.874442:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827655.874456:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827655.874461:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827655.874466:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790305792/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827655.874472:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4292 00000100:00080000:3.0:1787827655.874479:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827655.874480:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827655.874487:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787827655.874490:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827655.874496:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790305792:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787827655.874500:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827655.874515:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790306816/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827655.874522:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790305920:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827655.874550:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827655.874562:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827655/real 1787827655] req@ffff9e6831aa8000 x1874668790305920/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827655.874572:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790305920/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827655.874587:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827655.874589:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827655.874592:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790305920:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827655.874596:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790306048:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827655.874610:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827655.874617:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827655/real 1787827655] req@ffff9e6831aa8b40 x1874668790306048/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827655.874624:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790306048/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827655.874629:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827655.874632:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827655.874634:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790306048:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827655.874637:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790306176:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827655.874648:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827655.874653:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827655/real 1787827655] req@ffff9e6831aaad00 x1874668790306176/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827655.874661:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790306176/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827655.874667:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827655.874669:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827655.874671:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790306176:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827655.874674:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790306304:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827655.874684:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827655.874689:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827655/real 1787827655] req@ffff9e6831aa9a40 x1874668790306304/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827655.874722:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790306304/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827655.874731:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827655.874735:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827655.874738:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790306304:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827655.874746:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790306432:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827655.874759:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827655.874767:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827655/real 1787827655] req@ffff9e6831aaa1c0 x1874668790306432/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827655.874775:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790306432/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827655.874789:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827655.874792:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827655.874795:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790306432:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827655.874800:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790306560:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827655.874812:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827655.874819:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827655/real 1787827655] req@ffff9e6831aa8780 x1874668790306560/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827655.874827:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790306560/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827655.874832:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827655.874834:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827655.874836:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790306560:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827655.874839:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790306688:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827655.874850:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827655.874855:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827655/real 1787827655] req@ffff9e6831aa8f00 x1874668790306688/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827655.874862:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790306688/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827655.874867:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827655.874869:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827655.874871:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790306688:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827655.874888:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790306816:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827655.874898:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827655.874904:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827655/real 1787827655] req@ffff9e6831aaa940 x1874668790306816/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827655.874912:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790306816/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827655.874917:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827655.874918:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827655.874920:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790306816:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827656.385773:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827656.577888:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827657.409809:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827657.601833:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827658.433822:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827658.625888:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827659.457772:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827659.649837:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827660.481766:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827660.673871:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827660.993770:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827660.993779:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827660.993783:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827660.993791:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4297 00000100:00080000:1.0:1787827660.993800:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827660.993805:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827660.993839:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790306944/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827660.993870:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827660.993875:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827660.993877:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827660.993882:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4297 00000100:00080000:1.0:1787827660.993885:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827660.993890:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827660.993900:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827660.993921:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790306944:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787827660.993932:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790307072/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827660.993947:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827660.993951:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827660.993953:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827660.993958:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4297 00000100:00080000:1.0:1787827660.993963:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827660.993980:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827660.993986:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787827660.993998:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827660.994001:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790307200/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827660.994014:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827660.994018:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827660.994021:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827660.994025:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4297 00000100:00080000:1.0:1787827660.994029:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827660.994034:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827660.994040:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827660.994042:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827660/real 1787827660] req@ffff9e6831aaa580 x1874668790306944/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827660.994057:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790307328/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827660.994066:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790306944/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827660.994072:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827660.994082:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827660.994086:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827660.994088:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827660.994093:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4297 00000100:00080000:3.0:1787827660.994098:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787827660.994101:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827660.994105:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790306944:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827660.994110:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827660.994118:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827660.994131:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790307456/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827660.994137:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790307072:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827660.994152:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827660.994157:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827660.994159:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827660.994165:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4297 00000100:00080000:1.0:1787827660.994169:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787827660.994174:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827660.994177:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827660.994184:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827660.994192:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827660/real 1787827660] req@ffff9e6831aa9e00 x1874668790307072/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827660.994200:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790307584/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827660.994216:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827660.994216:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790307072/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827660.994225:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827660.994229:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827660.994239:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4297 00000100:00080000:3.0:1787827660.994239:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827660.994245:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827660.994248:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787827660.994253:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827660.994256:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790307072:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787827660.994260:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827660.994266:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790307200:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827660.994276:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790307712/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827660.994288:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787827660.994289:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827660.994295:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827660.994298:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827660.994303:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827660/real 1787827660] req@ffff9e6831aabc00 x1874668790307200/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827660.994304:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4297 00000100:00080000:1.0:1787827660.994312:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827660.994321:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827660.994326:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790307200/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787827660.994329:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787827660.994343:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827660.994347:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827660.994351:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790307840/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827660.994354:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790307200:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827660.994367:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790307328:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787827660.994373:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827660.994378:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827660.994381:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827660.994387:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4297 00000100:00080000:1.0:1787827660.994392:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787827660.994392:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827660.994400:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827660.994407:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827660.994407:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827660/real 1787827660] req@ffff9e6831aa9680 x1874668790307328/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827660.994424:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790307968/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827660.994440:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790307328/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827660.994452:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827660.994455:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827660.994460:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790307328:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827660.994475:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790307456:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827660.994492:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827660.994501:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827660/real 1787827660] req@ffff9e6831aaa940 x1874668790307456/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827660.994522:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790307456/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827660.994531:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827660.994535:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827660.994538:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790307456:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827660.994543:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790307584:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827660.994560:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827660.994570:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827660/real 1787827660] req@ffff9e6831aa8f00 x1874668790307584/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827660.994582:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790307584/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827660.994598:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827660.994601:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827660.994604:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790307584:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827660.994610:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790307712:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827660.994630:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827660.994640:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827660/real 1787827660] req@ffff9e6831aa8780 x1874668790307712/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827660.994652:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790307712/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827660.994661:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827660.994664:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827660.994667:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790307712:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827660.994673:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790307840:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827660.994688:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827660.994718:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827660/real 1787827660] req@ffff9e6831aaa1c0 x1874668790307840/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827660.994732:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790307840/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827660.994739:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827660.994743:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827660.994747:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790307840:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827660.994753:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790307968:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827660.994771:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827660.994782:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827660/real 1787827660] req@ffff9e6831aa9a40 x1874668790307968/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827660.994794:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790307968/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827660.994803:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827660.994806:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827660.994809:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790307968:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827661.505802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827661.697827:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827662.529762:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827662.721882:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827663.553756:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827663.745895:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827664.577783:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827664.769832:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827665.601791:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827665.793847:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827666.113818:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827666.113828:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827666.113833:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827666.113844:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4302 00000100:00080000:3.0:1787827666.113856:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827666.113863:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827666.113897:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790308096/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827666.113921:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827666.113924:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827666.113925:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827666.113928:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4302 00000100:00080000:3.0:1787827666.113941:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827666.113945:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827666.113953:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827666.113962:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790308224/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827666.113970:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790308096:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827666.113973:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827666.113977:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827666.113979:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827666.113982:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4302 00000100:00080000:3.0:1787827666.113984:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827666.113986:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827666.113990:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827666.113997:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790308352/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827666.114004:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827666.114006:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827666.114008:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827666.114010:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4302 00000100:00080000:3.0:1787827666.114020:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827666.114023:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827666.114026:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827666.114029:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827666.114037:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790308480/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827666.114049:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827666.114051:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827666.114052:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827666.114055:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4302 00000100:00080000:3.0:1787827666.114058:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827666.114060:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827666.114064:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827666.114065:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827666/real 1787827666] req@ffff9e680a2d70c0 x1874668790308096/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827666.114075:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790308608/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827666.114082:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827666.114085:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790308096/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827666.114086:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827666.114089:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827666.114096:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4302 00000100:00080000:3.0:1787827666.114101:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827666.114106:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827666.114106:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827666.114112:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787827666.114116:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827666.114120:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790308096:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787827666.114124:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790308736/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827666.114132:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827666.114134:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827666.114135:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827666.114138:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4302 00000100:00080000:3.0:1787827666.114141:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827666.114144:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790308224:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827666.114145:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827666.114152:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827666.114162:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790308864/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827666.114169:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827666.114172:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827666.114174:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787827666.114178:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827666.114178:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4302 00000100:00080000:3.0:1787827666.114184:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827666.114188:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827666.114191:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827666.114195:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827666/real 1787827666] req@ffff9e680a2d7c00 x1874668790308224/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827666.114200:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790308992/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827666.114218:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827666.114221:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790308224/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827666.114222:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827666.114225:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827666.114232:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4302 00000100:00080000:3.0:1787827666.114237:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827666.114243:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827666.114249:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787827666.114251:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827666.114257:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827666.114259:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790309120/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827666.114264:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790308224:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827666.114275:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790308352:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827666.114294:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827666.114303:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827666/real 1787827666] req@ffff9e680a2d43c0 x1874668790308352/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827666.114311:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790308352/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827666.114316:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827666.114319:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827666.114321:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790308352:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827666.114324:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790308480:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827666.114333:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827666.114339:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827666/real 1787827666] req@ffff9e680a2d6580 x1874668790308480/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827666.114346:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790308480/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827666.114351:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827666.114352:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827666.114355:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790308480:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827666.114359:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790308608:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827666.114368:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827666.114373:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827666/real 1787827666] req@ffff9e680a2d6d00 x1874668790308608/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827666.114380:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790308608/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827666.114385:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827666.114387:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827666.114388:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790308608:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827666.114392:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790308736:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827666.114401:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827666.114406:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827666/real 1787827666] req@ffff9e680a2d52c0 x1874668790308736/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827666.114413:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790308736/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827666.114419:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827666.114421:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827666.114422:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790308736:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827666.114435:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790308864:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827666.114445:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827666.114452:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827666/real 1787827666] req@ffff9e680a2d7480 x1874668790308864/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827666.114459:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790308864/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827666.114464:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827666.114465:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827666.114467:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790308864:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827666.114470:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790308992:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827666.114480:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827666.114485:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827666/real 1787827666] req@ffff9e680a2d4f00 x1874668790308992/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827666.114492:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790308992/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827666.114497:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827666.114499:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827666.114501:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790308992:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827666.114504:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790309120:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827666.114513:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827666.114518:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827666/real 1787827666] req@ffff9e680a2d4b40 x1874668790309120/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827666.114525:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790309120/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827666.114530:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827666.114532:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827666.114534:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790309120:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827666.625762:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827666.817862:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827667.649779:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827667.841855:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827668.673774:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827668.865868:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827669.697778:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827669.889815:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827670.721774:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827670.913909:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827671.233797:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827671.233810:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827671.233814:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827671.233825:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4307 00000100:00080000:3.0:1787827671.233838:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827671.233845:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827671.233882:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790309248/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827671.233924:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827671.233929:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827671.233931:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827671.233936:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4307 00000100:00080000:3.0:1787827671.233948:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827671.233953:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827671.233963:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827671.233967:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790309248:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787827671.233980:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790309376/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827671.233994:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827671.233998:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827671.234000:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827671.234004:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4307 00000100:00080000:3.0:1787827671.234010:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827671.234015:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787827671.234022:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787827671.234022:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827671.234039:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790309504/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827671.234050:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827671.234054:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827671.234057:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827671.234061:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827671/real 1787827671] req@ffff9e680a2d5a40 x1874668790309248/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827671.234064:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4307 00000100:00080000:3.0:1787827671.234072:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827671.234081:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827671.234084:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790309248/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787827671.234088:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787827671.234095:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827671.234099:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827671.234106:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790309248:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787827671.234107:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790309632/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827671.234126:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790309376:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827671.234129:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827671.234135:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827671.234138:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827671.234144:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4307 00000800:00000100:2.0:1787827671.234148:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827671.234151:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827671.234158:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827671.234161:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827671/real 1787827671] req@ffff9e680a2d4000 x1874668790309376/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787827671.234166:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827671.234180:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790309376/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827671.234186:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790309760/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827671.234189:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827671.234196:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827671.234203:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790309376:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827671.234209:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827671.234212:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790309504:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827671.234219:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827671.234223:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827671.234229:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4307 00000800:00000100:2.0:1787827671.234233:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827671.234241:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827671.234242:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827671/real 1787827671] req@ffff9e680a2d61c0 x1874668790309504/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827671.234252:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827671.234257:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790309504/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787827671.234261:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787827671.234267:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827671.234271:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827671.234274:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790309504:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827671.234281:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790309888/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827671.234287:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790309632:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787827671.234303:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787827671.234309:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827671.234309:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827671.234314:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827671.234318:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827671/real 1787827671] req@ffff9e680a2d5e00 x1874668790309632/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827671.234321:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4307 00000100:00080000:3.0:1787827671.234329:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827671.234334:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790309632/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827671.234338:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827671.234348:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787827671.234351:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827671.234355:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827671.234358:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790309632:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827671.234366:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790309760:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827671.234367:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790310016/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827671.234380:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827671.234382:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827671.234388:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787827671.234389:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827671/real 1787827671] req@ffff9e680a2d4780 x1874668790309760/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827671.234398:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790309760/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827671.234398:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827671.234409:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4307 00000100:00080000:2.0:1787827671.234411:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827671.234416:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827671.234417:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827671.234423:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790309760:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787827671.234427:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827671.234434:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827671.234437:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790309888:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827671.234454:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790310144/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827671.234457:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827671.234472:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827671.234475:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827671/real 1787827671] req@ffff9e680a2d5a40 x1874668790309888/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827671.234476:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827671.234481:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827671.234491:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4307 00000100:00100000:2.0:1787827671.234494:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790309888/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827671.234497:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827671.234506:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787827671.234508:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827671.234514:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:3.0:1787827671.234515:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827671.234519:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790309888:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827671.234526:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790310016:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827671.234531:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790310272/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827671.234542:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827671.234554:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827671/real 1787827671] req@ffff9e680a2d4b40 x1874668790310016/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827671.234562:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790310016/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827671.234568:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827671.234570:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827671.234572:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790310016:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827671.234580:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790310144:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827671.234590:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827671.234596:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827671/real 1787827671] req@ffff9e680a2d4f00 x1874668790310144/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827671.234603:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790310144/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827671.234608:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827671.234610:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827671.234611:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790310144:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827671.234615:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790310272:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827671.234624:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827671.234630:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827671/real 1787827671] req@ffff9e680a2d7480 x1874668790310272/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827671.234637:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790310272/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827671.234642:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827671.234644:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827671.234646:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790310272:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827671.745754:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827671.937882:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827672.769785:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827672.961869:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827673.793778:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827673.985844:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827674.817768:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827675.009838:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827675.841789:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827676.033964:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827676.353763:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827676.353774:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827676.353778:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827676.353789:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4313 00000100:00080000:3.0:1787827676.353802:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827676.353809:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827676.353851:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790310400/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827676.353902:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827676.353907:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827676.353910:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827676.353915:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4313 00000100:00080000:3.0:1787827676.353920:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827676.353925:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827676.353937:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827676.353955:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790310528/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827676.353970:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827676.353977:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787827676.353979:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790310400:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827676.353980:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827676.353988:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4313 00000100:00080000:3.0:1787827676.353994:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827676.354000:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827676.354005:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827676.354016:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790310656/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827676.354024:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827676.354027:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827676.354028:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827676.354032:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4313 00000100:00080000:3.0:1787827676.354036:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827676.354040:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827676.354046:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827676.354054:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790310784/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827676.354060:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827676.354063:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827676.354064:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787827676.354074:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827676.354081:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4313 00000100:00080000:3.0:1787827676.354088:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827676.354095:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827676.354101:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827676.354120:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790310912/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827676.354125:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827676/real 1787827676] req@ffff9e680a2d52c0 x1874668790310400/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827676.354172:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827676.354186:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827676.354189:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827676.354197:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4313 00000100:00100000:2.0:1787827676.354200:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790310400/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827676.354203:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827676.354218:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787827676.354222:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787827676.354225:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787827676.354231:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827676.354236:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790310400:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787827676.354246:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790311040/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827676.354264:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790310528:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827676.354289:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827676.354295:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827676.354299:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787827676.354307:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827676.354316:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4313 00000100:00100000:2.0:1787827676.354324:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827676/real 1787827676] req@ffff9e680a2d6d00 x1874668790310528/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827676.354326:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827676.354346:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827676.354350:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790310528/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787827676.354359:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787827676.354363:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827676.354367:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827676.354370:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790310528:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827676.354377:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790310656:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827676.354385:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790311168/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827676.354398:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787827676.354407:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827676.354408:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827676.354417:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827676.354425:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4313 00000100:00080000:3.0:1787827676.354431:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827676.354438:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827676/real 1787827676] req@ffff9e680a2d6580 x1874668790310656/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827676.354444:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827676.354456:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827676.354472:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790310656/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827676.354474:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790311296/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827676.354531:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827676.354532:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827676.354543:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827676.354544:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787827676.354552:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790310656:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827676.354552:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827676.354575:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4313 00000100:00100000:2.0:1787827676.354576:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790310784:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787827676.354590:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827676.354599:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827676.354607:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827676.354608:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827676.354620:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827676/real 1787827676] req@ffff9e680a2d43c0 x1874668790310784/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827676.354630:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790311424/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827676.354677:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790310784/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827676.354686:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827676.354690:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827676.354693:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790310784:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827676.354733:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790310912:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827676.354748:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827676.354756:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827676/real 1787827676] req@ffff9e680a2d7c00 x1874668790310912/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827676.354764:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790310912/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827676.354771:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827676.354773:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827676.354776:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790310912:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827676.354791:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790311040:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827676.354803:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827676.354809:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827676/real 1787827676] req@ffff9e680a2d70c0 x1874668790311040/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827676.354817:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790311040/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827676.354830:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827676.354832:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827676.354834:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790311040:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827676.354838:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790311168:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827676.354849:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827676.354856:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827676/real 1787827676] req@ffff9e680a2d52c0 x1874668790311168/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827676.354887:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790311168/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827676.354893:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827676.354895:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827676.354897:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790311168:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827676.354902:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790311296:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827676.354920:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827676.354928:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827676/real 1787827676] req@ffff9e680a2d7480 x1874668790311296/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827676.354936:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790311296/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827676.354942:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827676.354944:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827676.354947:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790311296:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827676.354951:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790311424:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827676.354963:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827676.354969:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827676/real 1787827676] req@ffff9e680a2d4f00 x1874668790311424/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827676.354977:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790311424/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827676.354982:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827676.354984:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827676.354986:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790311424:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827676.865813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827677.057824:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827677.889876:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827678.081934:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827678.913755:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827679.105920:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827679.937823:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827680.129869:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827680.961763:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827681.153861:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827681.473809:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827681.473835:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827681.473840:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827681.473850:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4318 00000100:00080000:3.0:1787827681.473861:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827681.473869:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827681.473917:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790311552/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827681.473947:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827681.473951:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827681.473953:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827681.473958:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4318 00000100:00080000:3.0:1787827681.473962:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827681.473967:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827681.473978:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827681.473992:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790311680/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827681.473996:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790311552:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827681.474012:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827681.474017:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827681.474019:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827681.474024:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4318 00000100:00080000:3.0:1787827681.474029:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827681.474034:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827681.474050:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827681.474065:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827681.474076:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790311808/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827681.474089:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827681.474093:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827681.474095:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827681.474099:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4318 00000100:00100000:2.0:1787827681.474102:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827681/real 1787827681] req@ffff9e680a2d4b40 x1874668790311552/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827681.474106:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827681.474116:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827681.474133:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827681.474136:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790311552/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827681.474149:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790311936/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827681.474151:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827681.474161:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827681.474167:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827681.474170:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790311552:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827681.474174:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827681.474179:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827681.474184:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4318 00000100:00080000:3.0:1787827681.474189:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827681.474193:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827681.474199:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790311680:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787827681.474199:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827681.474219:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790312064/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827681.474230:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827681.474232:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827681.474238:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827681.474240:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827681.474245:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4318 00000100:00100000:2.0:1787827681.474251:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827681/real 1787827681] req@ffff9e680a2d5a40 x1874668790311680/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827681.474251:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827681.474261:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827681.474270:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827681.474276:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790311680/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827681.474284:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827681.474286:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827681.474288:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790311680:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827681.474297:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790311808:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827681.474299:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790312192/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827681.474312:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787827681.474317:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827681.474319:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827681.474323:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827681.474329:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827681/real 1787827681] req@ffff9e680a2d4780 x1874668790311808/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827681.474330:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4318 00000100:00080000:3.0:1787827681.474338:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827681.474346:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790311808/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827681.474348:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827681.474358:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787827681.474359:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827681.474365:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827681.474368:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790311808:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827681.474373:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790311936:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827681.474379:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790312320/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827681.474388:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827681.474397:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827681.474400:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827681/real 1787827681] req@ffff9e680a2d5e00 x1874668790311936/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827681.474404:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827681.474409:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827681.474415:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790311936/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827681.474417:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4318 00000100:00080000:3.0:1787827681.474426:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827681.474434:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827681.474440:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827681.474442:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827681.474443:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790311936:192.168.203.109@tcp:8:kworker.0 00000008:00100000:3.0:1787827681.474453:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827681.474462:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790312064:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827681.474471:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790312448/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827681.474480:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827681.474492:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827681/real 1787827681] req@ffff9e680a2d61c0 x1874668790312064/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827681.474500:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827681.474510:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827681.474514:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827681.474515:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790312064/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827681.474525:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4318 00000100:00080000:2.0:1787827681.474528:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827681.474532:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827681.474533:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827681.474539:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790312064:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787827681.474543:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827681.474547:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790312192:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787827681.474553:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827681.474563:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827681.474571:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790312576/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827681.474579:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827681/real 1787827681] req@ffff9e680a2d4000 x1874668790312192/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827681.474604:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790312192/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827681.474615:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827681.474617:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827681.474620:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790312192:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827681.474626:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790312320:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827681.474646:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827681.474658:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827681/real 1787827681] req@ffff9e680a2d4b40 x1874668790312320/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827681.474683:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790312320/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827681.474688:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827681.474690:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827681.474692:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790312320:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827681.474725:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790312448:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827681.474756:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827681.474768:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827681/real 1787827681] req@ffff9e680a2d4f00 x1874668790312448/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827681.474780:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790312448/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827681.474796:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827681.474799:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827681.474812:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790312448:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827681.474816:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790312576:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827681.474851:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827681.474859:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827681/real 1787827681] req@ffff9e680a2d7480 x1874668790312576/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827681.474869:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790312576/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827681.474884:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827681.474887:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827681.474890:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790312576:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827681.985757:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827682.177856:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827683.009775:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827683.201885:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827684.033778:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827684.225866:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827685.057809:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827685.249886:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827686.081775:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827686.273847:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827686.593850:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827686.593858:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827686.593862:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827686.593881:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4323 00000100:00080000:0.0:1787827686.593891:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827686.593897:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827686.593930:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790312704/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827686.593955:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827686.593958:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827686.593959:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827686.593962:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4323 00000100:00080000:0.0:1787827686.593965:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827686.593967:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827686.593974:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827686.593984:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790312832/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827686.593998:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827686.593998:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790312704:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787827686.594011:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827686.594013:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827686.594016:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4323 00000100:00080000:0.0:1787827686.594019:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827686.594022:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827686.594025:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827686.594035:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790312960/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827686.594042:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827686.594045:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827686.594047:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827686.594049:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4323 00000100:00080000:0.0:1787827686.594052:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827686.594055:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787827686.594059:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787827686.594062:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827686.594072:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790313088/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827686.594083:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827686.594086:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827686.594088:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827686.594091:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4323 00000100:00080000:0.0:1787827686.594095:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827686.594098:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827686/real 1787827686] req@ffff9e6803e9a940 x1874668790312704/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827686.594101:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827686.594109:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827686.594122:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790313216/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827686.594128:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790312704/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827686.594133:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827686.594141:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827686.594145:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827686.594147:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827686.594149:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4323 00000100:00080000:0.0:1787827686.594153:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827686.594154:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787827686.594158:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827686.594160:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790312704:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1787827686.594163:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827686.594173:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790313344/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827686.594179:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827686.594185:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787827686.594185:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790312832:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827686.594190:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827686.594198:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4323 00000100:00080000:0.0:1787827686.594203:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827686.594209:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827686.594210:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827686.594219:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827686.594224:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827686/real 1787827686] req@ffff9e6803e99680 x1874668790312832/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827686.594235:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790313472/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827686.594243:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790312832/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827686.594277:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827686.594284:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827686.594285:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827686.594292:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790312832:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827686.594293:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827686.594298:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827686.594303:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790312960:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827686.594304:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4323 00000100:00080000:0.0:1787827686.594311:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827686.594316:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827686.594320:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827686.594330:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827686.594334:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790313600/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827686.594343:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827686.594345:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827686/real 1787827686] req@ffff9e6803e99e00 x1874668790312960/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827686.594347:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827686.594351:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827686.594360:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4323 00000100:00080000:0.0:1787827686.594364:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827686.594367:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790312960/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827686.594370:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827686.594377:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827686.594388:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790313728/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827686.594389:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827686.594397:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827686.594404:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790312960:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827686.594412:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790313088:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827686.594427:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827686.594435:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827686/real 1787827686] req@ffff9e6803e98f00 x1874668790313088/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827686.594443:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790313088/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827686.594448:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827686.594450:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827686.594452:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790313088:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827686.594456:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790313216:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827686.594466:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827686.594472:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827686/real 1787827686] req@ffff9e6803e9b840 x1874668790313216/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827686.594479:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790313216/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827686.594484:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827686.594486:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827686.594488:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790313216:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827686.594492:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790313344:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827686.594501:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827686.594507:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827686/real 1787827686] req@ffff9e6803e9a580 x1874668790313344/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827686.594514:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790313344/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827686.594520:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827686.594521:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827686.594523:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790313344:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827686.594535:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790313472:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827686.594546:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827686.594552:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827686/real 1787827686] req@ffff9e6803e9a1c0 x1874668790313472/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827686.594559:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790313472/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827686.594564:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827686.594566:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827686.594568:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790313472:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827686.594571:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790313600:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827686.594580:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827686.594587:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827686/real 1787827686] req@ffff9e6803e9b480 x1874668790313600/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827686.594603:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790313600/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827686.594611:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827686.594613:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827686.594615:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790313600:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827686.594619:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790313728:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827686.594631:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827686.594643:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827686/real 1787827686] req@ffff9e6803e98780 x1874668790313728/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827686.594655:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790313728/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827686.594673:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827686.594677:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827686.594680:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790313728:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827687.105830:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827687.297905:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827688.129815:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827688.321953:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827689.153787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827689.345997:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827690.177815:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827690.369864:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827691.201751:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827691.393955:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827691.713832:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827691.713843:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827691.713848:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827691.713858:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4328 00000100:00080000:0.0:1787827691.713870:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827691.713879:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827691.713915:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790313856/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827691.713940:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827691.713943:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827691.713945:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827691.713948:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4328 00000100:00080000:0.0:1787827691.713952:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827691.713956:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827691.713965:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827691.713972:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790313856:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827691.713978:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790313984/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827691.713989:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827691.713991:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827691.713993:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827691.713996:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4328 00000100:00080000:0.0:1787827691.713999:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827691.714002:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827691.714005:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827691.714015:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790314112/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827691.714022:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827691.714025:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827691.714026:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827691.714030:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4328 00000800:00000100:1.0:1787827691.714034:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827691.714035:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827691.714041:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827691.714045:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827691.714053:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790314240/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827691.714060:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827691.714063:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827691.714064:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827691.714066:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4328 00000100:00080000:0.0:1787827691.714069:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827691.714072:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827691/real 1787827691] req@ffff9e6803e9ad00 x1874668790313856/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827691.714074:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827691.714081:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827691.714096:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790314368/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827691.714097:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790313856/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827691.714118:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827691.714122:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827691.714124:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827691.714125:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827691.714128:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4328 00000100:00080000:0.0:1787827691.714134:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827691.714138:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787827691.714141:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827691.714146:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827691.714149:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790313856:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827691.714156:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790314496/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827691.714163:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827691.714165:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827691.714166:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827691.714170:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4328 00000100:00080000:0.0:1787827691.714172:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827691.714174:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827691.714177:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827691.714184:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790313984:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827691.714185:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790314624/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827691.714199:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827691.714202:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827691.714203:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827691.714206:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4328 00000100:00080000:0.0:1787827691.714208:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827691.714210:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827691.714213:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827691.714220:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790314752/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827691.714226:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827691.714229:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827691.714231:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827691.714234:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4328 00000100:00080000:0.0:1787827691.714239:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827691.714240:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827691.714244:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827691.714248:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827691.714258:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790314880/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827691.714267:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827691/real 1787827691] req@ffff9e6803e992c0 x1874668790313984/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827691.714288:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790313984/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827691.714307:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827691.714312:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827691.714318:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790313984:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827691.714327:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790314112:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827691.714356:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827691.714372:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827691/real 1787827691] req@ffff9e6803e9bc00 x1874668790314112/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827691.714385:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790314112/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827691.714394:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827691.714398:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827691.714403:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790314112:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827691.714411:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790314240:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827691.714431:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827691.714442:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827691/real 1787827691] req@ffff9e6803e98b40 x1874668790314240/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827691.714455:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790314240/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827691.714463:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827691.714467:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827691.714471:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790314240:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827691.714477:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790314368:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827691.714508:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827691.714521:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827691/real 1787827691] req@ffff9e6803e98780 x1874668790314368/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827691.714535:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790314368/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827691.714544:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827691.714548:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827691.714551:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790314368:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827691.714558:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790314496:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827691.714588:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827691.714600:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827691/real 1787827691] req@ffff9e6803e9b480 x1874668790314496/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827691.714614:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790314496/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827691.714624:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827691.714628:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827691.714631:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790314496:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827691.714651:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790314624:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827691.714673:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827691.714686:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827691/real 1787827691] req@ffff9e6803e9a1c0 x1874668790314624/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827691.714728:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790314624/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827691.714738:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827691.714742:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827691.714746:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790314624:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827691.714769:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790314752:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827691.714791:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827691.714807:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827691/real 1787827691] req@ffff9e6803e9a580 x1874668790314752/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827691.714821:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790314752/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827691.714829:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827691.714833:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827691.714836:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790314752:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827691.714842:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790314880:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827691.714861:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827691.714872:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827691/real 1787827691] req@ffff9e6803e9b840 x1874668790314880/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827691.714884:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790314880/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827691.714892:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827691.714896:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827691.714899:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790314880:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827692.225774:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827692.417901:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827693.249781:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827693.441916:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827694.273795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827694.465857:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827695.297841:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827695.489970:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827696.321778:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827696.513918:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827696.833817:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827696.833827:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827696.833830:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827696.833838:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4333 00000100:00080000:3.0:1787827696.833848:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827696.833853:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827696.833886:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790315008/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827696.833909:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827696.833913:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827696.833915:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827696.833917:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4333 00000100:00080000:3.0:1787827696.833920:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827696.833923:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827696.833931:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827696.833941:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790315136/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827696.833947:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790315008:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827696.833953:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827696.833957:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827696.833958:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827696.833962:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4333 00000100:00080000:3.0:1787827696.833964:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827696.833967:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827696.833970:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827696.833987:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790315264/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827696.833995:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827696.833998:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827696.834000:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827696.834002:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4333 00000100:00080000:3.0:1787827696.834005:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827696.834010:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1787827696.834012:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787827696.834019:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827696.834029:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790315392/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827696.834036:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827696.834039:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827696.834040:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827696.834043:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4333 00000100:00080000:3.0:1787827696.834046:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827696.834049:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827696.834052:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827696/real 1787827696] req@ffff9e680a2d52c0 x1874668790315008/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787827696.834054:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827696.834073:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790315520/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827696.834074:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790315008/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827696.834104:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827696.834106:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827696.834108:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827696.834112:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827696.834113:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827696.834123:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790315008:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827696.834123:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4333 00000100:00080000:3.0:1787827696.834131:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827696.834135:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827696.834139:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827696.834146:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790315648/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827696.834151:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790315136:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827696.834160:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827696.834164:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827696.834166:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827696.834170:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4333 00000100:00080000:3.0:1787827696.834173:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827696.834176:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827696.834179:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827696.834186:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827696.834190:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790315776/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827696.834199:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827696.834201:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827696.834203:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827696.834208:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4333 00000100:00100000:1.0:1787827696.834210:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827696/real 1787827696] req@ffff9e680a2d70c0 x1874668790315136/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827696.834211:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827696.834217:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827696.834222:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827696.834234:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790315904/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827696.834239:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790315136/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827696.834245:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827696.834255:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827696.834258:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827696.834259:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827696.834265:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4333 00000100:00080000:1.0:1787827696.834266:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827696.834272:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790315136:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827696.834273:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827696.834279:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790315264:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827696.834282:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827696.834290:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787827696.834303:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827696.834311:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790316032/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827696.834316:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827696/real 1787827696] req@ffff9e680a2d7c00 x1874668790315264/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827696.834342:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790315264/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827696.834348:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827696.834351:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827696.834355:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790315264:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827696.834360:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790315392:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827696.834372:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827696.834378:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827696/real 1787827696] req@ffff9e680a2d43c0 x1874668790315392/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827696.834386:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790315392/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827696.834392:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827696.834394:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827696.834396:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790315392:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827696.834400:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790315520:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827696.834410:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827696.834416:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827696/real 1787827696] req@ffff9e680a2d6580 x1874668790315520/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827696.834424:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790315520/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827696.834429:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827696.834431:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827696.834434:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790315520:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827696.834454:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790315648:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827696.834477:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827696.834490:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827696/real 1787827696] req@ffff9e680a2d6d00 x1874668790315648/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827696.834501:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790315648/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827696.834508:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827696.834510:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827696.834513:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790315648:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827696.834517:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790315776:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827696.834532:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827696.834540:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827696/real 1787827696] req@ffff9e680a2d52c0 x1874668790315776/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827696.834552:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790315776/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827696.834558:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827696.834560:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827696.834562:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790315776:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827696.834565:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790315904:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827696.834576:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827696.834581:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827696/real 1787827696] req@ffff9e680a2d7480 x1874668790315904/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827696.834589:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790315904/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827696.834594:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827696.834596:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827696.834598:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790315904:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827696.834601:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790316032:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827696.834611:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827696.834617:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827696/real 1787827696] req@ffff9e680a2d4f00 x1874668790316032/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827696.834624:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790316032/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827696.834633:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827696.834635:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827696.834638:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790316032:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827697.345799:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827697.537890:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827698.369811:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827698.561861:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827699.393832:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827699.585871:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827700.417774:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827700.609906:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827701.441784:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827701.633916:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827701.953800:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827701.953808:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827701.953812:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827701.953820:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4338 00000100:00080000:1.0:1787827701.953830:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827701.953836:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827701.953870:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790316160/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827701.953896:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827701.953900:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827701.953901:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827701.953904:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4338 00000100:00080000:1.0:1787827701.953907:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827701.953909:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827701.953917:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827701.953927:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790316288/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827701.953935:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827701.953938:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827701.953939:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827701.953942:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4338 00000100:00080000:1.0:1787827701.953945:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827701.953947:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827701.953950:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827701.953955:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790316160:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787827701.953961:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790316416/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827701.953969:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827701.953972:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827701.953973:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827701.953976:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4338 00000100:00080000:1.0:1787827701.953979:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827701.953982:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827701.953985:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827701.954007:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790316544/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827701.954017:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827701.954021:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827701.954024:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827701.954028:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4338 00000100:00080000:1.0:1787827701.954034:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827701.954040:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827701.954048:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827701.954053:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790316288:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827701.954064:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790316672/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827701.954074:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827701.954077:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827701.954079:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827701.954082:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4338 00000100:00080000:1.0:1787827701.954085:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827701.954092:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827701.954093:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790316416:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787827701.954098:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827701.954112:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790316800/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827701.954121:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827701.954123:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790316544:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787827701.954127:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827701.954131:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827701.954135:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4338 00000100:00080000:1.0:1787827701.954138:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827701.954142:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827701.954146:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827701.954152:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790316672:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827701.954158:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790316928/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827701.954168:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827701.954171:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827701.954173:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827701.954177:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4338 00000100:00100000:2.0:1787827701.954181:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790316800:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827701.954182:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827701.954189:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827701.954195:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827701.954207:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790317056/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827701.954212:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790316928:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827701.954225:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827701.954229:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827701.954231:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827701.954235:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4338 00000100:00080000:1.0:1787827701.954239:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827701.954242:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827701.954246:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790317056:192.168.203.109@tcp:8:kworker.0 00000008:00100000:1.0:1787827701.954248:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827701.954264:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790317184/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827701.954279:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790317184:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827702.465796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827702.657833:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827703.489772:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827703.681879:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827704.513832:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827704.705877:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787827705.026033:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:0.0:1787827705.026078:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787827705.026095:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827705.026143:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:0.0:1787827705.026156:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827705.026167:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827705.026175:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827705.026185:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827705.026194:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787827705.026209:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827701/real 1787827705] req@ffff9e6831aaa940 x1874668790317184/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787827705.026212:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827705.026227:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787827705.026290:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790317184/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827705.026309:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827705.026316:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827705.026324:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790317184:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827705.026336:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827701/real 1787827705] req@ffff9e6831aa8f00 x1874668790317056/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827705.026354:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790317056/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827705.026367:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827705.026374:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827705.026394:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790317056:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827705.026405:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827701/real 1787827705] req@ffff9e6831aaa1c0 x1874668790316800/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827705.026424:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790316800/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827705.026439:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827705.026444:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827705.026449:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790316800:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827705.026463:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827701/real 1787827705] req@ffff9e6831aa8780 x1874668790316928/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827705.026479:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790316928/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827705.026490:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827705.026494:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827705.026500:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790316928:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827705.026508:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827701/real 1787827705] req@ffff9e6831aa8000 x1874668790316416/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827705.026524:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790316416/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827705.026536:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827705.026542:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827705.026546:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790316416:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827705.026556:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827701/real 1787827705] req@ffff9e6831aab480 x1874668790316544/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827705.026574:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790316544/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827705.026587:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827705.026590:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827705.026594:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790316544:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827705.026605:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827701/real 1787827705] req@ffff9e6831aa9a40 x1874668790316672/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827705.026623:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790316672/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827705.026634:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827705.026639:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827705.026644:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790316672:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827705.026653:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827701/real 1787827705] req@ffff9e6831aaad00 x1874668790316160/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827705.026734:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790316160/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827705.026749:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827705.026755:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827705.026761:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790316160:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787827705.026772:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827701/real 1787827705] req@ffff9e6831aa8b40 x1874668790316288/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827705.026790:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790316288/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827705.026803:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827705.026808:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827705.026813:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790316288:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827705.537778:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827705.729832:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827706.561795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827706.753800:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827707.073864:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827707.073874:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827707.073877:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827707.073886:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4343 00000100:00080000:3.0:1787827707.073895:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827707.073902:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827707.073933:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790317312/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827707.073956:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827707.073959:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827707.073961:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827707.073963:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4343 00000100:00080000:3.0:1787827707.073966:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827707.073969:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827707.073978:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827707.073989:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790317312:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787827707.073990:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790317440/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827707.074003:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827707.074005:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827707.074007:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827707.074010:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4343 00000100:00080000:3.0:1787827707.074012:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827707.074014:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827707.074018:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827707.074026:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790317568/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827707.074033:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827707.074035:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827707.074036:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827707.074039:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4343 00000100:00080000:3.0:1787827707.074042:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827707.074044:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827707.074047:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827707.074054:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790317696/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827707.074057:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827707.074065:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827707.074068:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827707.074070:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827707.074074:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4343 00000100:00080000:3.0:1787827707.074076:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827707.074078:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827707.074082:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827707.074094:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790317824/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827707.074098:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827707/real 1787827707] req@ffff9e680a2d4b40 x1874668790317312/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827707.074144:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827707.074150:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827707.074153:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827707.074159:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790317312/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827707.074164:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4343 00000100:00080000:3.0:1787827707.074175:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827707.074186:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827707.074187:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787827707.074200:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:3.0:1787827707.074201:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827707.074224:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790317312:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787827707.074232:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790317952/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827707.074243:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827707.074246:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827707.074249:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827707.074253:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790317440:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827707.074258:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4343 00000100:00080000:3.0:1787827707.074264:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827707.074268:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827707.074274:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827707.074289:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827707.074291:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790318080/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827707.074304:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827707.074308:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827707/real 1787827707] req@ffff9e680a2d4000 x1874668790317440/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827707.074309:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827707.074313:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827707.074322:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4343 00000100:00080000:3.0:1787827707.074329:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827707.074336:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790317440/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827707.074336:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827707.074345:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787827707.074353:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827707.074359:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827707.074362:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790318208/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827707.074367:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790317440:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827707.074381:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790317568:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827707.074383:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827707.074392:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827707.074394:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827707.074400:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4343 00000100:00080000:3.0:1787827707.074407:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827707.074414:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827707.074414:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827707.074422:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827707.074432:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827707/real 1787827707] req@ffff9e680a2d61c0 x1874668790317568/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827707.074438:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790318336/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827707.074458:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790317568/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827707.074468:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827707.074473:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827707.074477:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790317568:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827707.074485:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790317696:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827707.074506:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827707.074518:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827707/real 1787827707] req@ffff9e680a2d5e00 x1874668790317696/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827707.074529:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790317696/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827707.074537:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827707.074540:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827707.074544:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790317696:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827707.074550:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790317824:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827707.074569:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827707.074580:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827707/real 1787827707] req@ffff9e680a2d4780 x1874668790317824/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827707.074599:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790317824/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827707.074608:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827707.074611:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827707.074615:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790317824:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827707.074621:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790317952:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827707.074642:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827707.074669:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827707/real 1787827707] req@ffff9e680a2d5a40 x1874668790317952/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827707.074680:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790317952/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827707.074690:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827707.074717:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827707.074721:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790317952:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827707.074741:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790318080:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827707.074761:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827707.074772:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827707/real 1787827707] req@ffff9e680a2d4b40 x1874668790318080/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827707.074782:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790318080/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827707.074803:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827707.074806:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827707.074809:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790318080:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827707.074814:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790318208:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827707.074832:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827707.074841:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827707/real 1787827707] req@ffff9e680a2d4f00 x1874668790318208/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827707.074851:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790318208/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827707.074858:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827707.074861:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827707.074863:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790318208:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827707.074869:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790318336:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827707.074885:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827707.074895:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827707/real 1787827707] req@ffff9e680a2d7480 x1874668790318336/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827707.074905:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790318336/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827707.074912:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827707.074915:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827707.074918:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790318336:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827707.585807:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827707.777875:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827708.609811:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827708.801884:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827709.633774:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827709.825840:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827710.657786:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827710.849872:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827711.681760:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827711.873865:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827712.193769:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827712.193777:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827712.193780:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827712.193788:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4348 00000100:00080000:3.0:1787827712.193798:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827712.193804:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827712.193839:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790318464/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827712.193867:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827712.193872:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827712.193874:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827712.193879:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4348 00000100:00080000:3.0:1787827712.193884:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827712.193889:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827712.193897:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827712.193904:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790318464:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787827712.193911:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790318592/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827712.193945:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827712.193948:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827712.193950:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827712.193954:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4348 00000100:00080000:3.0:1787827712.193957:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827712.193960:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827712.193964:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827712.193974:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790318720/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827712.193983:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827712.194002:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827712.194005:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827712.194009:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4348 00000100:00080000:3.0:1787827712.194014:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827712.194018:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787827712.194026:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787827712.194029:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827712.194044:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790318848/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827712.194054:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827712.194058:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827712.194060:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827712.194064:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4348 00000100:00080000:3.0:1787827712.194068:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827712.194072:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827712.194094:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827712.194096:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827712/real 1787827712] req@ffff9e680a2d52c0 x1874668790318464/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827712.194108:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790318976/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827712.194119:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827712.194125:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790318464/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827712.194126:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827712.194131:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827712.194137:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4348 00000100:00080000:3.0:1787827712.194157:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827712.194160:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827712.194164:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827712.194178:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790319104/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827712.194190:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827712.194204:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827712.194224:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827712.194226:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827712.194231:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827712.194239:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4348 00000100:00100000:2.0:1787827712.194240:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790318464:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827712.194242:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827712.194250:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827712.194260:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827712.194279:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790319232/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827712.194287:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790318592:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827712.194311:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827712.194318:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827712.194322:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827712.194329:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4348 00000100:00080000:3.0:1787827712.194336:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827712.194344:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787827712.194345:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787827712.194353:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827712.194388:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827712/real 1787827712] req@ffff9e680a2d6d00 x1874668790318592/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827712.194388:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790319360/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827712.194422:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827712.194426:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827712.194428:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827712.194431:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790318592/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827712.194436:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4348 00000100:00080000:3.0:1787827712.194444:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827712.194470:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827712.194470:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827712.194475:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787827712.194479:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827712.194486:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790318592:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827712.194492:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790319488/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827712.194497:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790318720:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827712.194556:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827712.194571:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827712/real 1787827712] req@ffff9e680a2d6580 x1874668790318720/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827712.194583:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790318720/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827712.194592:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827712.194595:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827712.194598:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790318720:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827712.194604:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790318848:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827712.194661:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827712.194669:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827712/real 1787827712] req@ffff9e680a2d43c0 x1874668790318848/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827712.194677:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790318848/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827712.194683:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827712.194685:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827712.194688:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790318848:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827712.194692:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790318976:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827712.194725:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827712.194732:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827712/real 1787827712] req@ffff9e680a2d7c00 x1874668790318976/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827712.194740:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790318976/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827712.194746:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827712.194748:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827712.194750:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790318976:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827712.194754:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790319104:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827712.194764:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827712.194770:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827712/real 1787827712] req@ffff9e680a2d70c0 x1874668790319104/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827712.194777:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790319104/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827712.194783:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827712.194785:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827712.194787:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790319104:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827712.194800:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790319232:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827712.194812:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827712.194818:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827712/real 1787827712] req@ffff9e680a2d7480 x1874668790319232/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827712.194830:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790319232/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827712.194835:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827712.194837:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827712.194839:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790319232:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827712.194843:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790319360:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827712.194852:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827712.194858:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827712/real 1787827712] req@ffff9e680a2d4f00 x1874668790319360/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827712.194865:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790319360/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827712.194870:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827712.194872:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827712.194874:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790319360:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827712.194877:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790319488:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827712.194887:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827712.194892:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827712/real 1787827712] req@ffff9e680a2d4b40 x1874668790319488/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827712.194899:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790319488/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827712.194914:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827712.194917:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827712.194920:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790319488:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827712.705771:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827712.897926:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827713.729777:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827713.921937:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827714.753788:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827714.945854:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827715.777818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827715.969972:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827716.801810:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827716.993873:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827717.313764:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827717.313771:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827717.313775:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827717.313783:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4354 00000100:00080000:3.0:1787827717.313793:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827717.313800:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827717.313839:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790319616/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827717.313865:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827717.313870:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827717.313872:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827717.313876:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4354 00000100:00080000:3.0:1787827717.313881:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827717.313885:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827717.313894:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827717.313907:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790319744/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827717.313912:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790319616:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827717.313919:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827717.313921:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827717.313923:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827717.313927:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4354 00000100:00080000:3.0:1787827717.313930:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827717.313933:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827717.313937:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827717.313945:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790319872/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827717.313953:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827717.313956:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827717.313957:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827717.313961:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4354 00000100:00080000:3.0:1787827717.313964:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827717.313969:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827717.313977:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827717.313982:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827717.313991:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790320000/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827717.314000:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827717.314003:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827717.314004:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827717.314008:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827717/real 1787827717] req@ffff9e680a2d5a40 x1874668790319616/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827717.314008:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4354 00000100:00080000:3.0:1787827717.314012:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827717.314015:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827717.314020:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827717.314025:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790319616/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827717.314029:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790320128/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827717.314036:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827717.314037:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827717.314041:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827717.314041:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827717.314042:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827717.314045:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790319616:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827717.314046:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4354 00000100:00080000:3.0:1787827717.314049:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827717.314052:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827717.314056:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827717.314064:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790320256/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827717.314066:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790319744:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827717.314074:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827717.314077:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827717.314079:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827717.314082:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4354 00000100:00080000:3.0:1787827717.314085:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827717.314087:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827717.314089:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827717.314093:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827717.314098:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827717/real 1787827717] req@ffff9e680a2d4780 x1874668790319744/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827717.314101:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790320384/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827717.314118:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827717.314119:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790319744/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827717.314121:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827717.314123:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827717.314127:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4354 00000100:00080000:2.0:1787827717.314129:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827717.314130:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827717.314132:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827717.314134:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827717.314136:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790319744:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787827717.314138:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827717.314142:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790319872:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827717.314147:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790320512/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827717.314156:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787827717.314158:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827717.314160:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827717.314162:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827717.314165:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4354 00000100:00100000:2.0:1787827717.314168:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827717/real 1787827717] req@ffff9e680a2d5e00 x1874668790319872/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827717.314168:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827717.314172:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827717.314176:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827717.314180:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790319872/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827717.314185:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790320640/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827717.314188:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827717.314192:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827717.314195:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790319872:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827717.314201:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790320000:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827717.314219:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827717.314231:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827717/real 1787827717] req@ffff9e680a2d61c0 x1874668790320000/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827717.314240:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790320000/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827717.314250:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827717.314252:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827717.314254:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790320000:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827717.314258:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790320128:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827717.314271:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827717.314278:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827717/real 1787827717] req@ffff9e680a2d4000 x1874668790320128/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827717.314285:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790320128/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827717.314290:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827717.314292:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827717.314293:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790320128:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827717.314306:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790320256:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827717.314316:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827717.314322:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827717/real 1787827717] req@ffff9e680a2d5a40 x1874668790320256/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827717.314329:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790320256/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827717.314335:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827717.314337:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827717.314339:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790320256:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827717.314343:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790320384:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827717.314353:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827717.314358:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827717/real 1787827717] req@ffff9e680a2d4b40 x1874668790320384/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827717.314365:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790320384/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827717.314370:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827717.314372:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827717.314381:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790320384:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827717.314385:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790320512:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827717.314395:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827717.314401:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827717/real 1787827717] req@ffff9e680a2d4f00 x1874668790320512/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827717.314408:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790320512/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827717.314413:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827717.314415:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827717.314417:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790320512:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827717.314420:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790320640:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827717.314429:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827717.314435:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827717/real 1787827717] req@ffff9e680a2d7480 x1874668790320640/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827717.314442:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790320640/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827717.314447:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827717.314449:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827717.314451:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790320640:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827717.825787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827718.017908:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827718.849781:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827719.041818:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827719.873797:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827720.065910:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827720.897790:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827721.089875:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827721.921763:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827722.113918:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827722.433780:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827722.433789:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827722.433793:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827722.433802:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4359 00000100:00080000:3.0:1787827722.433811:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827722.433817:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827722.433849:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790320768/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827722.433868:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827722.433871:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827722.433872:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827722.433875:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4359 00000100:00080000:3.0:1787827722.433877:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827722.433880:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827722.433887:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827722.433895:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790320896/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827722.433902:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827722.433915:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827722.433916:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827722.433919:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4359 00000100:00080000:3.0:1787827722.433921:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827722.433923:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827722.433926:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827722.433933:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790320768:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787827722.433936:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790321024/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827722.433957:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827722.433960:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827722.433962:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827722.433967:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4359 00000100:00080000:3.0:1787827722.433971:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827722.433984:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827722.433989:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827722.434001:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790321152/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827722.434008:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827722.434010:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827722.434011:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827722.434014:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4359 00000100:00080000:3.0:1787827722.434017:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827722.434019:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827722.434022:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827722.434027:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827722.434035:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790321280/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827722.434042:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827722.434044:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827722.434045:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827722.434048:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4359 00000100:00080000:3.0:1787827722.434052:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827722.434055:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827722.434059:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827722.434063:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827722/real 1787827722] req@ffff9e680a2d70c0 x1874668790320768/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827722.434069:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790321408/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827722.434086:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827722.434089:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827722.434090:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827722.434093:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790320768/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827722.434096:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4359 00000100:00080000:3.0:1787827722.434102:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827722.434108:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827722.434114:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787827722.434116:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827722.434122:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827722.434124:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790321536/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827722.434129:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790320768:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827722.434140:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827722.434144:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787827722.434145:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790320896:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827722.434146:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827722.434154:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4359 00000100:00080000:3.0:1787827722.434159:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827722.434162:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827722.434166:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827722.434174:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790321664/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827722.434182:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827722.434194:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827722/real 1787827722] req@ffff9e680a2d7c00 x1874668790320896/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827722.434204:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827722.434210:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827722.434211:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827722.434217:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790320896/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827722.434217:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4359 00000100:00080000:3.0:1787827722.434223:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827722.434229:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827722.434235:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787827722.434243:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787827722.434246:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790321792/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827722.434253:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827722.434261:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790320896:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827722.434287:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790321024:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827722.434316:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827722.434324:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827722/real 1787827722] req@ffff9e680a2d43c0 x1874668790321024/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827722.434340:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790321024/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827722.434346:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827722.434348:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827722.434351:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790321024:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827722.434355:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790321152:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827722.434376:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827722.434382:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827722/real 1787827722] req@ffff9e680a2d6580 x1874668790321152/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827722.434395:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790321152/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827722.434400:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827722.434402:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827722.434404:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790321152:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827722.434407:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790321280:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827722.434417:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827722.434422:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827722/real 1787827722] req@ffff9e680a2d6d00 x1874668790321280/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827722.434430:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790321280/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827722.434435:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827722.434436:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827722.434438:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790321280:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827722.434441:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790321408:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827722.434450:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827722.434456:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827722/real 1787827722] req@ffff9e680a2d52c0 x1874668790321408/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827722.434463:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790321408/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827722.434473:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827722.434475:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827722.434477:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790321408:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827722.434480:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790321536:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827722.434490:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827722.434495:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827722/real 1787827722] req@ffff9e680a2d7480 x1874668790321536/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827722.434502:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790321536/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827722.434507:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827722.434509:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827722.434510:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790321536:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827722.434514:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790321664:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827722.434523:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827722.434528:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827722/real 1787827722] req@ffff9e680a2d4f00 x1874668790321664/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827722.434554:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790321664/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827722.434560:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827722.434562:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827722.434565:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790321664:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827722.434569:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790321792:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827722.434594:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827722.434601:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827722/real 1787827722] req@ffff9e680a2d4b40 x1874668790321792/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827722.434608:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790321792/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827722.434613:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827722.434615:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827722.434617:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790321792:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827722.945813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827723.137808:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827723.969755:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827724.161857:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827724.993819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827725.186741:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827726.017757:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827726.209805:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827727.041795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827727.233838:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827727.553789:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827727.553797:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827727.553801:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827727.553810:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4364 00000100:00080000:1.0:1787827727.553819:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827727.553826:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827727.553861:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790321920/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827727.553889:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827727.553892:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827727.553893:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827727.553897:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4364 00000100:00080000:1.0:1787827727.553900:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827727.553903:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827727.553912:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827727.553924:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790321920:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787827727.553925:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790322048/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827727.553938:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827727.553941:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827727.553943:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827727.553947:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4364 00000100:00080000:1.0:1787827727.553950:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827727.553953:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827727.553957:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827727.553966:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790322176/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827727.553978:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827727.553980:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827727.553982:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787827727.553985:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827727.553989:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4364 00000100:00080000:1.0:1787827727.553994:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827727.553997:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827727.554013:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827727.554023:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790322304/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827727.554029:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827727.554032:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827727.554034:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827727.554038:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827727/real 1787827727] req@ffff9e6831aa9680 x1874668790321920/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827727.554039:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4364 00000100:00080000:1.0:1787827727.554044:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827727.554050:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827727.554057:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827727.554066:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790321920/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827727.554079:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790322432/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827727.554086:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827727.554089:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827727.554090:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827727.554093:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4364 00000100:00080000:1.0:1787827727.554096:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827727.554100:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827727.554103:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827727.554111:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790322560/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827727.554117:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827727.554119:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827727.554121:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827727.554123:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4364 00000100:00080000:1.0:1787827727.554125:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827727.554127:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827727.554129:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827727.554134:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790322688/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827727.554140:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827727.554142:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827727.554143:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827727.554146:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4364 00000100:00080000:3.0:1787827727.554147:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827727.554148:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827727.554157:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:3.0:1787827727.554158:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:1.0:1787827727.554160:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827727.554166:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790321920:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787827727.554169:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790322816/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827727.554185:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827727.554187:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827727.554189:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827727.554191:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4364 00000100:00080000:1.0:1787827727.554193:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827727.554196:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827727.554199:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827727.554202:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790322048:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827727.554207:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790322944/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827727.554360:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827727.554376:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827727/real 1787827727] req@ffff9e6831aabc00 x1874668790322048/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827727.554390:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790322048/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827727.554398:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827727.554462:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827727.554468:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790322048:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827727.554476:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790322176:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827727.554501:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827727.554526:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827727/real 1787827727] req@ffff9e6831aa9e00 x1874668790322176/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827727.554562:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790322176/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827727.554567:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827727.554570:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827727.554572:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790322176:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827727.554577:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790322304:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827727.554592:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827727.554608:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827727/real 1787827727] req@ffff9e6831aaa580 x1874668790322304/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827727.554616:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790322304/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827727.554621:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827727.554623:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827727.554625:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790322304:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827727.554629:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790322432:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827727.554639:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827727.554644:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827727/real 1787827727] req@ffff9e6831aa8b40 x1874668790322432/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827727.554650:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790322432/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827727.554655:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827727.554658:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827727.554660:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790322432:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827727.554663:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790322560:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827727.554672:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827727.554677:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827727/real 1787827727] req@ffff9e6831aaad00 x1874668790322560/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827727.554684:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790322560/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827727.554690:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827727.554692:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827727.554694:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790322560:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827727.554698:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790322688:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827727.554732:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827727.554752:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827727/real 1787827727] req@ffff9e6831aa9a40 x1874668790322688/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827727.554766:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790322688/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827727.554775:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827727.554777:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827727.554779:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790322688:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827727.554784:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790322816:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827727.554800:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827727.554807:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827727/real 1787827727] req@ffff9e6831aab480 x1874668790322816/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827727.554814:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790322816/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827727.554820:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827727.554822:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827727.554824:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790322816:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827727.554840:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790322944:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827727.554850:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827727.554856:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827727/real 1787827727] req@ffff9e6831aa8000 x1874668790322944/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827727.554863:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790322944/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827727.554868:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827727.554870:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827727.554872:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790322944:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827728.065765:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827728.257891:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827729.089804:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827729.281818:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827730.113830:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827730.305837:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827731.137796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827731.329827:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827732.161777:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827732.353950:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827732.673762:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827732.673773:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827732.673778:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827732.673788:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4369 00000100:00080000:1.0:1787827732.673798:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827732.673804:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827732.673842:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790323072/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827732.673876:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827732.673879:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827732.673881:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827732.673884:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4369 00000100:00080000:1.0:1787827732.673888:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827732.673891:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827732.673899:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827732.673912:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790323200/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827732.673919:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790323072:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827732.673922:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827732.673927:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827732.673928:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827732.673931:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4369 00000100:00080000:1.0:1787827732.673933:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827732.673935:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827732.673939:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827732.673947:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790323328/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827732.673953:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827732.673956:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827732.673957:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827732.673959:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4369 00000100:00080000:1.0:1787827732.673962:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827732.673965:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827732.673968:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787827732.673979:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827732.673980:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790323456/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827732.673993:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827732.673995:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827732.673997:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827732.674000:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4369 00000100:00080000:1.0:1787827732.674002:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827732.674004:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827732.674006:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827732.674015:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790323584/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827732.674021:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827732/real 1787827732] req@ffff9e6831aa8780 x1874668790323072/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827732.674024:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827732.674032:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827732.674035:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827732.674040:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4369 00000100:00080000:1.0:1787827732.674042:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827732.674047:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827732.674047:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790323072/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787827732.674053:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827732.674062:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790323712/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827732.674066:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827732.674069:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827732.674072:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827732.674075:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827732.674075:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787827732.674083:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4369 00000100:00100000:3.0:1787827732.674086:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790323072:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827732.674087:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827732.674094:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827732.674099:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827732.674110:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790323840/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827732.674117:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790323200:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827732.674125:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827732.674131:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827732.674133:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827732.674138:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4369 00000800:00000100:3.0:1787827732.674143:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827732.674145:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827732.674152:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827732.674157:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827732.674157:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827732/real 1787827732] req@ffff9e6831aaa1c0 x1874668790323200/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827732.674172:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790323968/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827732.674179:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790323200/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827732.674182:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827732.674190:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827732.674193:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827732.674199:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4369 00000100:00080000:3.0:1787827732.674199:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827732.674201:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827732.674208:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787827732.674209:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827732.674216:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827732.674220:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790323200:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827732.674225:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790324096/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827732.674232:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790323328:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827732.674260:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827732.674270:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827732/real 1787827732] req@ffff9e6831aa8f00 x1874668790323328/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827732.674278:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790323328/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827732.674283:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827732.674285:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827732.674287:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790323328:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827732.674291:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790323456:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827732.674301:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827732.674306:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827732/real 1787827732] req@ffff9e6831aaa940 x1874668790323456/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827732.674313:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790323456/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827732.674323:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827732.674325:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827732.674327:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790323456:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827732.674330:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790323584:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827732.674339:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827732.674344:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827732/real 1787827732] req@ffff9e6831aa8000 x1874668790323584/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827732.674351:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790323584/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827732.674356:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827732.674358:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827732.674360:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790323584:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827732.674363:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790323712:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827732.674372:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827732.674378:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827732/real 1787827732] req@ffff9e6831aab480 x1874668790323712/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827732.674384:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790323712/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827732.674389:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827732.674391:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827732.674394:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790323712:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827732.674407:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790323840:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827732.674418:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827732.674423:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827732/real 1787827732] req@ffff9e6831aa9a40 x1874668790323840/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827732.674430:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790323840/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827732.674438:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827732.674441:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827732.674445:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790323840:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827732.674450:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790323968:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827732.674485:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827732.674499:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827732/real 1787827732] req@ffff9e6831aaad00 x1874668790323968/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827732.674507:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790323968/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827732.674513:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827732.674515:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827732.674517:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790323968:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827732.674521:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790324096:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827732.674535:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827732.674542:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827732/real 1787827732] req@ffff9e6831aa8b40 x1874668790324096/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827732.674549:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790324096/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827732.674554:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827732.674556:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827732.674558:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790324096:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827733.185839:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827733.377867:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827734.209748:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827734.401834:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827735.233747:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827735.425826:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827736.257781:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827736.449874:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827737.281788:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827737.473885:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827737.793812:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827737.793824:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827737.793828:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827737.793837:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4374 00000100:00080000:3.0:1787827737.793848:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827737.793856:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827737.793892:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790324224/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827737.793922:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827737.793925:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827737.793926:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827737.793929:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4374 00000100:00080000:3.0:1787827737.793933:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827737.793936:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827737.793945:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827737.793960:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790324352/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827737.793972:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827737.793975:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827737.793977:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827737.793982:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4374 00000100:00080000:3.0:1787827737.793986:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827737.793991:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827737.794001:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790324224:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787827737.794001:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827737.794029:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790324480/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827737.794044:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827737.794048:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827737.794051:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827737.794057:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4374 00000100:00080000:3.0:1787827737.794077:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827737.794082:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827737.794089:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827737.794095:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827737.794112:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790324608/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827737.794127:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827737.794130:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827737.794134:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827737.794140:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4374 00000100:00080000:3.0:1787827737.794145:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827737.794149:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827737/real 1787827737] req@ffff9e680a2d5a40 x1874668790324224/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827737.794155:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827737.794177:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790324224/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787827737.794183:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787827737.794188:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827737.794194:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827737.794199:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790324224:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787827737.794206:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790324736/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827737.794211:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790324352:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827737.794246:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827737.794251:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827737.794254:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827737.794261:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4374 00000800:00000100:2.0:1787827737.794268:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827737.794294:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827737.794299:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827737/real 1787827737] req@ffff9e680a2d4000 x1874668790324352/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827737.794307:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827737.794319:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790324352/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787827737.794320:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787827737.794336:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827737.794339:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827737.794358:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790324352:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827737.794361:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790324864/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827737.794375:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827737.794378:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827737.794382:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827737.794388:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4374 00000100:00080000:3.0:1787827737.794393:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827737.794397:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790324480:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827737.794406:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827737.794427:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827737.794477:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790324992/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827737.794482:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827737.794496:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827737/real 1787827737] req@ffff9e680a2d61c0 x1874668790324480/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827737.794500:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827737.794523:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787827737.794528:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790324480/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827737.794530:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827737.794548:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827737.794549:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4374 00000100:00080000:3.0:1787827737.794572:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827737.794575:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827737.794583:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790324480:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827737.794588:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827737.794595:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790324608:192.168.203.109@tcp:8:kworker.0 00000008:00100000:3.0:1787827737.794598:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827737.794622:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790325120/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827737.794625:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827737.794639:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827737/real 1787827737] req@ffff9e680a2d5e00 x1874668790324608/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827737.794643:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827737.794661:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827737.794666:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827737.794668:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790324608/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827737.794689:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4374 00000100:00080000:2.0:1787827737.794700:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827737.794706:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827737.794710:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790324608:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827737.794717:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790324736:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827737.794734:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827737.794754:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827737.794762:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827737.794770:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827737.794786:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790325248/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827737.794795:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827737/real 1787827737] req@ffff9e680a2d4780 x1874668790324736/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827737.794811:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790324736/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827737.794821:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827737.794826:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827737.794829:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790324736:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827737.794834:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790324864:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827737.794850:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827737.794857:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827737/real 1787827737] req@ffff9e680a2d4b40 x1874668790324864/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827737.794865:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790324864/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827737.794871:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827737.794873:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827737.794875:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790324864:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827737.794887:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790324992:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827737.794898:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827737.794904:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827737/real 1787827737] req@ffff9e680a2d4f00 x1874668790324992/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827737.794912:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790324992/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827737.794921:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827737.794923:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827737.794925:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790324992:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827737.794929:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790325120:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827737.794939:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827737.794944:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827737/real 1787827737] req@ffff9e680a2d7480 x1874668790325120/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827737.794951:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790325120/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827737.794956:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827737.794958:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827737.794960:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790325120:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827737.794963:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790325248:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827737.794973:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827737.794978:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827737/real 1787827737] req@ffff9e680a2d52c0 x1874668790325248/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827737.794985:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790325248/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827737.794990:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827737.794991:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827737.794994:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790325248:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827738.305824:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827738.497966:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827739.329912:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827739.521779:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827740.353771:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827740.545860:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827741.377757:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827741.569908:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827742.401858:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827742.593865:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827742.913811:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827742.913826:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827742.913831:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827742.913845:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4379 00000100:00080000:3.0:1787827742.913860:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827742.913870:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827742.913921:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790325376/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827742.913954:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827742.913958:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827742.913960:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827742.913964:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4379 00000100:00080000:3.0:1787827742.913968:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827742.913973:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827742.913985:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827742.914002:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790325504/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827742.914016:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827742.914028:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790325376:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827742.914033:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827742.914043:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827742.914051:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4379 00000100:00080000:3.0:1787827742.914057:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827742.914064:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827742.914082:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827742.914099:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790325632/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827742.914121:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827742.914125:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827742.914128:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827742.914133:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4379 00000800:00000100:2.0:1787827742.914140:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827742.914140:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827742.914164:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827742.914173:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827742.914191:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790325760/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827742.914206:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827742.914210:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827742.914213:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827742.914238:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4379 00000100:00080000:3.0:1787827742.914242:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827742.914249:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827742/real 1787827742] req@ffff9e680a2d6d00 x1874668790325376/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827742.914251:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827742.914263:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827742.914289:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790325888/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827742.914300:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790325376/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827742.914314:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827742.914331:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827742.914334:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827742.914336:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827742.914341:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827742.914351:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790325376:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827742.914352:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4379 00000100:00080000:3.0:1787827742.914361:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827742.914369:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827742.914379:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827742.914412:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790326016/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827742.914422:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790325504:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827742.914427:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827742.914431:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827742.914432:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827742.914450:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4379 00000100:00080000:3.0:1787827742.914457:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827742.914463:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827742.914471:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827742.914479:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827742.914495:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790326144/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827742.914501:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827742/real 1787827742] req@ffff9e680a2d6580 x1874668790325504/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827742.914548:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790325504/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827742.914548:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827742.914568:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827742.914576:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827742.914578:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827742.914589:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827742.914595:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4379 00000100:00100000:2.0:1787827742.914599:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790325504:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827742.914609:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790325632:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827742.914618:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827742.914629:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787827742.914636:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787827742.914640:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827742.914650:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827742/real 1787827742] req@ffff9e680a2d43c0 x1874668790325632/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827742.914666:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790326272/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827742.914667:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790325632/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827742.914703:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827742.914708:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827742.914742:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827742.914745:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827742.914747:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827742.914753:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4379 00000100:00080000:3.0:1787827742.914759:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827742.914765:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827742.914772:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827742.914779:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790325632:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827742.914789:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790325760:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827742.914808:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790326400/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827742.914849:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827742.914869:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827742/real 1787827742] req@ffff9e680a2d7c00 x1874668790325760/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827742.914886:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790325760/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827742.914899:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827742.914904:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827742.914908:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790325760:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827742.914916:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790325888:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827742.914945:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827742.914961:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827742/real 1787827742] req@ffff9e680a2d70c0 x1874668790325888/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827742.914978:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790325888/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827742.914990:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827742.914995:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827742.914999:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790325888:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827742.915027:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790326016:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827742.915051:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827742.915062:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827742/real 1787827742] req@ffff9e680a2d52c0 x1874668790326016/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827742.915074:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790326016/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827742.915088:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827742.915108:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827742.915113:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790326016:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827742.915121:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790326144:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827742.915148:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827742.915179:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827742/real 1787827742] req@ffff9e680a2d7480 x1874668790326144/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827742.915197:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790326144/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827742.915205:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827742.915208:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827742.915210:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790326144:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827742.915216:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790326272:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827742.915235:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827742.915248:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827742/real 1787827742] req@ffff9e680a2d4f00 x1874668790326272/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827742.915294:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790326272/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827742.915308:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827742.915313:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827742.915318:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790326272:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827742.915325:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790326400:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827742.915353:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827742.915364:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827742/real 1787827742] req@ffff9e680a2d4b40 x1874668790326400/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827742.915375:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790326400/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827742.915387:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827742.915390:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827742.915393:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790326400:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827743.425780:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:2.0:1787827743.617817:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827744.449805:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827744.641859:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827745.473749:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827745.665864:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827746.497777:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827746.689887:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827747.521763:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827747.713811:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827748.033773:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827748.033782:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827748.033786:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827748.033795:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4384 00000100:00080000:3.0:1787827748.033806:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827748.033813:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827748.033850:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790326528/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827748.033889:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827748.033893:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827748.033895:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827748.033900:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4384 00000100:00080000:3.0:1787827748.033904:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827748.033908:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827748.033918:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827748.033946:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790326656/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827748.033956:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827748.033959:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827748.033961:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827748.033965:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4384 00000100:00080000:3.0:1787827748.033968:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827748.033972:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827748.033977:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827748.033978:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790326528:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787827748.033991:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790326784/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827748.034003:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827748.034007:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827748.034009:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827748.034013:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4384 00000100:00080000:3.0:1787827748.034037:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827748.034042:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787827748.034047:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787827748.034049:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827748.034066:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790326912/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827748.034077:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827748.034081:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787827748.034084:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827748/real 1787827748] req@ffff9e680a2d4780 x1874668790326528/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827748.034085:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827748.034106:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4384 00000100:00080000:3.0:1787827748.034111:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827748.034112:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790326528/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827748.034121:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787827748.034127:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787827748.034130:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787827748.034132:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827748.034139:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790326528:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787827748.034145:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790326656:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827748.034149:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790327040/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827748.034161:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787827748.034166:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827748.034171:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827748.034174:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827748.034179:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827748/real 1787827748] req@ffff9e680a2d5e00 x1874668790326656/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827748.034180:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4384 00000100:00080000:3.0:1787827748.034188:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827748.034197:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827748.034204:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790326656/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787827748.034206:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787827748.034217:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827748.034221:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827748.034223:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790326656:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827748.034234:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790327168/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827748.034243:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790326784:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827748.034247:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827748.034252:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827748.034255:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787827748.034259:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827748.034264:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4384 00000100:00100000:2.0:1787827748.034269:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827748/real 1787827748] req@ffff9e680a2d61c0 x1874668790326784/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827748.034272:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827748.034282:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827748.034284:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790326784/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787827748.034290:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787827748.034292:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827748.034297:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827748.034300:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790326784:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827748.034304:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790326912:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827748.034309:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790327296/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827748.034319:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827748.034323:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827748.034330:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827748/real 1787827748] req@ffff9e680a2d4000 x1874668790326912/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827748.034338:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790326912/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827748.034342:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827748.034347:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827748.034355:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827748.034358:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827748.034373:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4384 00000100:00100000:2.0:1787827748.034379:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790326912:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787827748.034379:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827748.034389:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827748.034390:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790327040:192.168.203.109@tcp:8:kworker.0 00000008:00100000:3.0:1787827748.034401:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827748.034411:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827748.034421:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790327424/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827748.034422:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827748/real 1787827748] req@ffff9e680a2d5a40 x1874668790327040/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827748.034456:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790327040/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827748.034458:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827748.034467:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827748.034471:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827748.034471:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827748.034479:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827748.034479:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4384 00000100:00100000:2.0:1787827748.034486:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790327040:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787827748.034487:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827748.034494:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827748.034498:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827748.034501:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790327168:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827748.034511:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790327552/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827748.034518:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827748.034529:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827748/real 1787827748] req@ffff9e680a2d4b40 x1874668790327168/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827748.034538:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790327168/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827748.034545:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827748.034547:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827748.034550:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790327168:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827748.034554:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790327296:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827748.034563:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827748.034568:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827748/real 1787827748] req@ffff9e680a2d4f00 x1874668790327296/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827748.034575:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790327296/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827748.034580:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827748.034582:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827748.034584:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790327296:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827748.034587:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790327424:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827748.034596:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827748.034601:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827748/real 1787827748] req@ffff9e680a2d7480 x1874668790327424/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827748.034607:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790327424/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827748.034612:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827748.034614:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827748.034616:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790327424:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827748.034626:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790327552:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827748.034636:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827748.034642:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827748/real 1787827748] req@ffff9e680a2d52c0 x1874668790327552/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827748.034649:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790327552/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827748.034654:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827748.034656:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827748.034657:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790327552:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827748.545777:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827748.737868:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827749.569756:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827749.761910:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827750.593794:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827750.785839:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827751.617758:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827751.809877:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827752.641778:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827752.833846:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827753.153784:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827753.153792:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827753.153795:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827753.153803:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4389 00000100:00080000:1.0:1787827753.153812:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827753.153818:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827753.153850:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790327680/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827753.153872:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827753.153874:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827753.153876:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827753.153879:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4389 00000100:00080000:1.0:1787827753.153881:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827753.153883:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827753.153890:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827753.153899:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790327808/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827753.153905:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827753.153908:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827753.153909:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827753.153912:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4389 00000100:00080000:1.0:1787827753.153914:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827753.153916:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827753.153919:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827753.153925:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790327936/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827753.153931:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827753.153933:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827753.153934:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827753.153937:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4389 00000100:00080000:1.0:1787827753.153939:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827753.153942:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827753.153945:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827753.153950:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790327680:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787827753.153956:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790328064/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827753.153963:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827753.153965:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827753.153966:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827753.153969:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4389 00000100:00080000:1.0:1787827753.153972:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827753.153974:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827753.153977:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827753.153986:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790328192/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827753.153991:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827753.153993:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827753.153994:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827753.153997:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4389 00000100:00080000:1.0:1787827753.154001:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827753.154004:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827753.154007:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827753.154013:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827753.154016:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790328320/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827753.154023:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827753.154025:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827753.154027:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827753.154029:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4389 00000100:00080000:1.0:1787827753.154038:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827753.154041:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827753.154043:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827753.154050:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790328448/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827753.154056:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827753.154060:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827753.154062:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827753.154063:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827753/real 1787827753] req@ffff9e6831aaa580 x1874668790327680/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827753.154067:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4389 00000100:00080000:1.0:1787827753.154071:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827753.154077:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827753.154082:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827753.154091:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790328576/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827753.154094:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790327680/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827753.154108:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827753.154112:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827753.154114:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827753.154115:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827753.154119:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4389 00000100:00080000:1.0:1787827753.154122:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827753.154123:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787827753.154127:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827753.154130:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790327680:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787827753.154132:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827753.154141:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790327808:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827753.154143:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790328704/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827753.154175:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827753.154186:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827753/real 1787827753] req@ffff9e6831aa9e00 x1874668790327808/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827753.154195:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790327808/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827753.154200:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827753.154203:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827753.154205:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790327808:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827753.154209:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790327936:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827753.154221:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827753.154227:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827753/real 1787827753] req@ffff9e6831aabc00 x1874668790327936/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827753.154234:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790327936/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827753.154239:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827753.154241:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827753.154242:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790327936:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827753.154260:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790328064:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827753.154271:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827753.154276:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827753/real 1787827753] req@ffff9e6831aa9680 x1874668790328064/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827753.154284:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790328064/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827753.154290:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827753.154291:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827753.154293:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790328064:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827753.154296:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790328192:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827753.154306:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827753.154311:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827753/real 1787827753] req@ffff9e6831aa8b40 x1874668790328192/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827753.154317:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790328192/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827753.154323:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827753.154339:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827753.154341:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790328192:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827753.154346:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790328320:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827753.154360:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827753.154368:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827753/real 1787827753] req@ffff9e6831aaad00 x1874668790328320/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827753.154375:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790328320/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827753.154381:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827753.154383:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827753.154386:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790328320:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827753.154389:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790328448:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827753.154400:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827753.154405:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827753/real 1787827753] req@ffff9e6831aa9a40 x1874668790328448/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827753.154412:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790328448/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827753.154417:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827753.154419:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827753.154421:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790328448:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827753.154424:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790328576:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827753.154433:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827753.154438:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827753/real 1787827753] req@ffff9e6831aab480 x1874668790328576/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827753.154444:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790328576/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827753.154449:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827753.154451:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827753.154453:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790328576:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827753.154456:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790328704:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827753.154465:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827753.154481:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827753/real 1787827753] req@ffff9e6831aa8000 x1874668790328704/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827753.154488:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790328704/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827753.154493:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827753.154494:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827753.154496:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790328704:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827753.665779:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827753.857989:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827754.689811:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827754.881906:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827755.713749:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827755.905864:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827756.737766:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827756.929833:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827757.761806:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827757.953913:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827758.273780:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827758.273789:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827758.273793:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827758.273801:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4394 00000100:00080000:3.0:1787827758.273810:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827758.273815:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827758.273848:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790328832/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827758.273870:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827758.273872:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827758.273874:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827758.273876:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4394 00000100:00080000:3.0:1787827758.273878:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827758.273881:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827758.273888:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827758.273896:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790328960/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827758.273904:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827758.273907:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827758.273908:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827758.273911:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4394 00000100:00080000:3.0:1787827758.273913:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827758.273918:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827758.273920:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790328832:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787827758.273922:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827758.273932:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790329088/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827758.273939:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827758.273941:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827758.273942:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827758.273945:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4394 00000100:00080000:3.0:1787827758.273947:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827758.273950:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827758.273953:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827758.273961:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790329216/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827758.273967:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827758.273970:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827758.273971:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827758.273973:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4394 00000100:00080000:3.0:1787827758.273976:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827758.273978:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787827758.273984:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787827758.273986:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827758.273996:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790329344/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827758.274003:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827758.274005:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827758.274007:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827758.274010:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4394 00000100:00080000:3.0:1787827758.274012:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827758.274015:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827758.274019:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827758.274024:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827758/real 1787827758] req@ffff9e680a2d70c0 x1874668790328832/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827758.274029:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790329472/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827758.274057:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827758.274059:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827758.274060:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827758.274064:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4394 00000100:00080000:3.0:1787827758.274066:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827758.274069:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790328832/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827758.274072:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827758.274077:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827758.274086:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790329600/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827758.274090:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827758.274095:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827758.274098:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827758.274101:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827758.274101:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827758.274110:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4394 00000100:00100000:2.0:1787827758.274113:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790328832:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827758.274114:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827758.274121:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827758.274125:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827758.274129:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790328960:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827758.274134:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790329728/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827758.274141:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827758.274143:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827758.274145:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827758.274148:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4394 00000100:00080000:3.0:1787827758.274152:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827758.274156:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787827758.274160:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787827758.274163:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827758.274175:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790329856/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827758.274177:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827758/real 1787827758] req@ffff9e680a2d7c00 x1874668790328960/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827758.274203:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790328960/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827758.274210:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827758.274214:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827758.274217:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790328960:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827758.274237:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790329088:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827758.274250:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827758.274257:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827758/real 1787827758] req@ffff9e680a2d43c0 x1874668790329088/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827758.274265:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790329088/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827758.274270:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827758.274272:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827758.274274:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790329088:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827758.274277:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790329216:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827758.274287:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827758.274310:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827758/real 1787827758] req@ffff9e680a2d6580 x1874668790329216/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827758.274318:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790329216/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827758.274324:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827758.274326:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827758.274328:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790329216:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827758.274331:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790329344:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827758.274343:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827758.274349:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827758/real 1787827758] req@ffff9e680a2d6d00 x1874668790329344/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827758.274356:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790329344/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827758.274361:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827758.274363:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827758.274365:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790329344:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827758.274368:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790329472:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827758.274378:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827758.274384:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827758/real 1787827758] req@ffff9e680a2d52c0 x1874668790329472/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827758.274391:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790329472/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827758.274397:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827758.274398:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827758.274401:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790329472:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827758.274404:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790329600:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827758.274413:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827758.274419:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827758/real 1787827758] req@ffff9e680a2d7480 x1874668790329600/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827758.274425:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790329600/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827758.274430:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827758.274431:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827758.274434:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790329600:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827758.274437:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790329728:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827758.274447:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827758.274452:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827758/real 1787827758] req@ffff9e680a2d4f00 x1874668790329728/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827758.274459:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790329728/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827758.274464:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827758.274466:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827758.274468:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790329728:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827758.274471:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790329856:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827758.274480:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827758.274486:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827758/real 1787827758] req@ffff9e680a2d4b40 x1874668790329856/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827758.274493:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790329856/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827758.274498:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827758.274500:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827758.274502:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790329856:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827758.785777:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827758.977866:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827759.809823:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827760.001972:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827760.833745:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827761.025865:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827761.857782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827762.049905:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827762.881779:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827763.073883:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827763.393778:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827763.393786:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827763.393791:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827763.393800:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4400 00000100:00080000:3.0:1787827763.393809:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827763.393814:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827763.393845:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790329984/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827763.393874:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827763.393879:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827763.393881:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827763.393885:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4400 00000100:00080000:3.0:1787827763.393889:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827763.393893:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827763.393903:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827763.393919:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790330112/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827763.393927:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790329984:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827763.393929:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827763.393932:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827763.393934:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827763.393938:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4400 00000100:00080000:3.0:1787827763.393941:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827763.393946:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827763.393951:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827763.393964:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790330240/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827763.393973:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827763.393977:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827763.393979:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827763.393982:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4400 00000100:00080000:3.0:1787827763.393986:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787827763.393988:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827763.393990:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827763.393995:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827763.394005:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790330368/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827763.394015:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827763.394018:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827763.394020:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827763.394025:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4400 00000100:00100000:0.0:1787827763.394026:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827763/real 1787827763] req@ffff9e680a2d5a40 x1874668790329984/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827763.394029:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827763.394033:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827763.394039:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827763.394046:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790329984/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827763.394050:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790330496/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827763.394065:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827763.394070:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827763.394076:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790329984:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827763.394103:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790330112:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827763.394130:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827763.394137:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827763.394141:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827763.394144:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827763.394148:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4400 00000100:00100000:0.0:1787827763.394149:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827763/real 1787827763] req@ffff9e680a2d4000 x1874668790330112/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827763.394153:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827763.394157:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827763.394161:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790330112/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787827763.394163:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787827763.394170:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827763.394173:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827763.394175:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790330624/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827763.394177:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790330112:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827763.394183:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790330240:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827763.394202:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827763.394214:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827763/real 1787827763] req@ffff9e680a2d61c0 x1874668790330240/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827763.394226:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790330240/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827763.394234:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827763.394236:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827763.394239:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790330240:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827763.394269:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790330368:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827763.394292:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827763.394305:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827763/real 1787827763] req@ffff9e680a2d5e00 x1874668790330368/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827763.394310:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827763.394314:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827763.394316:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787827763.394319:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790330368/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827763.394320:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4400 00000100:00080000:3.0:1787827763.394324:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827763.394329:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827763.394332:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827763.394336:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790330368:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827763.394361:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790330496:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787827763.394372:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827763.394378:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787827763.394386:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827763.394390:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790330752/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827763.394398:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827763/real 1787827763] req@ffff9e680a2d4780 x1874668790330496/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827763.394400:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827763.394404:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827763.394405:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827763.394409:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4400 00000100:00100000:0.0:1787827763.394410:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790330496/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827763.394413:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827763.394417:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787827763.394419:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787827763.394421:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787827763.394422:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827763.394425:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790330496:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827763.394430:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790330880/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827763.394431:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790330624:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827763.394440:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827763.394443:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827763.394445:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787827763.394447:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827763.394449:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4400 00000100:00080000:3.0:1787827763.394452:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827763.394455:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827763.394458:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827763/real 1787827763] req@ffff9e680a2d5a40 x1874668790330624/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787827763.394460:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827763.394469:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790331008/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827763.394470:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790330624/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827763.394481:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827763.394484:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827763.394487:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790330624:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827763.394500:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790330752:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787827763.394527:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827763.394537:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827763/real 1787827763] req@ffff9e680a2d4b40 x1874668790330752/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827763.394545:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790330752/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827763.394551:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827763.394553:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827763.394555:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790330752:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827763.394558:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790330880:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827763.394571:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827763.394577:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827763/real 1787827763] req@ffff9e680a2d4f00 x1874668790330880/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827763.394584:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790330880/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827763.394593:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827763.394595:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827763.394597:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790330880:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827763.394600:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790331008:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787827763.394609:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827763.394615:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827763/real 1787827763] req@ffff9e680a2d7480 x1874668790331008/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827763.394621:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790331008/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827763.394626:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827763.394628:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827763.394630:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790331008:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827763.905790:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827764.097910:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827764.929782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827765.121891:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827765.953758:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827766.145920:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827766.977788:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827767.169840:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827768.001783:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827768.193832:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827768.513776:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827768.513785:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827768.513789:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827768.513813:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4405 00000100:00080000:3.0:1787827768.513822:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827768.513828:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827768.513862:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790331136/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827768.513882:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827768.513886:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827768.513887:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827768.513899:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4405 00000100:00080000:3.0:1787827768.513902:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827768.513907:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827768.513917:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827768.513929:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790331264/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827768.513930:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790331136:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827768.513941:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827768.513944:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827768.513946:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827768.513949:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4405 00000100:00080000:3.0:1787827768.513953:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827768.513956:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827768.513976:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827768.513985:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790331392/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827768.513994:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827768.513997:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827768.513998:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827768.514002:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4405 00000100:00080000:3.0:1787827768.514012:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827768.514017:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827768.514022:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827768.514033:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790331520/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827768.514043:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790331264:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827768.514043:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827768.514047:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827768.514049:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827768.514053:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4405 00000100:00080000:3.0:1787827768.514057:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827768.514061:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827768.514066:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827768.514076:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790331392:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827768.514076:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790331648/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827768.514086:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827768.514105:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827768.514108:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827768.514112:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4405 00000100:00080000:3.0:1787827768.514116:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827768.514117:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790331520:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787827768.514121:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827768.514126:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827768.514145:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790331648:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827768.514146:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790331776/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827768.514157:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827768.514161:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827768.514163:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827768.514182:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4405 00000100:00080000:3.0:1787827768.514185:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787827768.514186:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790331776:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827768.514189:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827768.514194:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827768.514206:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790331904/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827768.514240:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827768.514243:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827768.514245:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827768.514249:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4405 00000100:00080000:3.0:1787827768.514253:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827768.514257:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827768.514260:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790331904:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787827768.514261:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827768.514273:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790332032/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827768.514283:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827768.514286:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787827768.514287:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790332032:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787827768.514288:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827768.514292:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4405 00000100:00080000:3.0:1787827768.514296:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827768.514300:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827768.514304:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827768.514314:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790332160/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787827768.514359:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790332160:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827769.025768:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827769.217869:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827770.049847:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827770.241873:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827771.073772:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827771.265863:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787827771.585920:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:1.0:1787827771.585964:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787827771.585980:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787827771.586025:0:2387:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:1.0:1787827771.586037:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787827771.586045:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787827771.586053:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787827771.586062:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787827771.586071:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787827771.586081:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787827771.586091:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1787827771.586092:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827768/real 1787827771] req@ffff9e680a2d4b40 x1874668790332160/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827771.586120:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790332160/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827771.586133:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827771.586138:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827771.586142:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790332160:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827771.586150:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827768/real 1787827771] req@ffff9e680a2d4f00 x1874668790332032/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827771.586163:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790332032/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827771.586170:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827771.586174:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827771.586177:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790332032:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827771.586184:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827768/real 1787827771] req@ffff9e680a2d7480 x1874668790331904/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827771.586215:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790331904/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827771.586224:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827771.586228:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827771.586231:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790331904:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827771.586237:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827768/real 1787827771] req@ffff9e680a2d70c0 x1874668790331776/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827771.586262:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790331776/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827771.586272:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827771.586275:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827771.586278:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790331776:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827771.586285:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827768/real 1787827771] req@ffff9e680a2d43c0 x1874668790331520/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827771.586297:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790331520/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827771.586305:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827771.586309:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827771.586312:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790331520:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827771.586319:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827768/real 1787827771] req@ffff9e680a2d7c00 x1874668790331648/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827771.586330:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790331648/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827771.586339:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827771.586342:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827771.586345:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790331648:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827771.586351:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827768/real 1787827771] req@ffff9e680a2d6d00 x1874668790331264/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827771.586362:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790331264/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827771.586370:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827771.586374:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827771.586377:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790331264:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827771.586383:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827768/real 1787827771] req@ffff9e680a2d6580 x1874668790331392/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827771.586394:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790331392/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827771.586403:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827771.586407:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827771.586410:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790331392:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827771.586416:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827768/real 1787827771] req@ffff9e680a2d52c0 x1874668790331136/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827771.586429:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790331136/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827771.586438:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827771.586441:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827771.586444:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790331136:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787827772.097769:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827772.289909:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827773.121802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827773.314010:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827773.633827:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827773.633837:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827773.633841:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827773.633848:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4410 00000100:00080000:0.0:1787827773.633857:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827773.633863:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827773.633897:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790332288/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827773.633926:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827773.633931:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827773.633932:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827773.633936:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4410 00000100:00080000:0.0:1787827773.633940:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827773.633944:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827773.633955:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827773.633971:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790332416/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827773.633983:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827773.633986:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827773.633989:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827773.633998:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4410 00000100:00100000:2.0:1787827773.634000:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790332288:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787827773.634003:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827773.634015:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827773.634022:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827773.634036:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790332544/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827773.634047:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827773.634052:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827773.634054:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827773.634057:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4410 00000100:00080000:0.0:1787827773.634062:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827773.634068:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827773.634072:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827773.634083:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790332672/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827773.634101:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787827773.634110:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827773.634112:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827773.634116:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827773.634123:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4410 00000100:00080000:0.0:1787827773.634128:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827773.634134:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827773.634142:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827773.634162:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790332800/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827773.634175:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827773.634178:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827773.634181:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827773.634197:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4410 00000100:00100000:2.0:1787827773.634204:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827773/real 1787827773] req@ffff9e6803e98f00 x1874668790332288/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827773.634209:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827773.634231:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827773.634241:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827773.634251:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790332288/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827773.634259:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790332928/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827773.634291:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827773.634294:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827773.634302:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827773.634304:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787827773.634307:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827773.634315:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790332288:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787827773.634322:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4410 00000100:00080000:0.0:1787827773.634330:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827773.634336:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827773.634344:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827773.634358:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790333056/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827773.634366:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790332416:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827773.634390:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827773.634394:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827773.634397:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827773.634405:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4410 00000100:00080000:0.0:1787827773.634412:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827773.634420:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827773.634428:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827773.634439:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827773.634450:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827773/real 1787827773] req@ffff9e6803e99e00 x1874668790332416/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827773.634461:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790333184/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827773.634481:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790332416/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827773.634482:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827773.634506:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827773.634514:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827773.634519:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827773.634525:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4410 00000100:00080000:2.0:1787827773.634532:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787827773.634535:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827773.634543:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790332416:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827773.634551:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827773.634562:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827773.634565:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790332544:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827773.634586:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790333312/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827773.634602:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827773.634624:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827773/real 1787827773] req@ffff9e6803e99680 x1874668790332544/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827773.634641:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790332544/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827773.634650:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827773.634656:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827773.634659:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790332544:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827773.634669:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790332672:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827773.634741:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827773.634757:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827773/real 1787827773] req@ffff9e6803e9a940 x1874668790332672/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827773.634769:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790332672/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827773.634778:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827773.634782:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827773.634786:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790332672:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827773.634793:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790332800:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827773.634816:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827773.634829:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827773/real 1787827773] req@ffff9e6803e9b840 x1874668790332800/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827773.634841:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790332800/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827773.634851:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827773.634855:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827773.634858:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790332800:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827773.634865:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790332928:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827773.634887:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827773.634900:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827773/real 1787827773] req@ffff9e6803e9a580 x1874668790332928/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827773.634914:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790332928/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827773.634924:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827773.634928:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827773.634932:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790332928:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827773.634957:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790333056:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827773.634980:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827773.634994:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827773/real 1787827773] req@ffff9e6803e9a1c0 x1874668790333056/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827773.635007:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790333056/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827773.635015:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827773.635019:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827773.635023:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790333056:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827773.635030:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790333184:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827773.635063:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827773.635079:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827773/real 1787827773] req@ffff9e6803e9b480 x1874668790333184/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827773.635092:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790333184/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827773.635106:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827773.635110:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827773.635113:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790333184:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827773.635120:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790333312:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827773.635142:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827773.635154:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827773/real 1787827773] req@ffff9e6803e98780 x1874668790333312/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827773.635165:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790333312/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827773.635173:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827773.635176:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827773.635180:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790333312:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827774.145845:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827774.337952:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827775.169833:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827775.362084:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827776.193768:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827776.385848:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827777.217803:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827777.409906:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827778.241750:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827778.433903:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827778.753824:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827778.753833:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827778.753838:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827778.753848:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4415 00000100:00080000:3.0:1787827778.753859:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827778.753866:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827778.753914:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790333440/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827778.753942:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827778.753946:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827778.753949:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827778.753953:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4415 00000100:00080000:3.0:1787827778.753957:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827778.753962:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827778.753981:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827778.753994:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790333568/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827778.754006:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827778.754010:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790333440:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827778.754010:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827778.754015:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827778.754024:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4415 00000100:00080000:3.0:1787827778.754028:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827778.754042:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827778.754048:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827778.754060:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790333696/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827778.754071:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827778.754075:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827778.754076:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827778.754081:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4415 00000100:00080000:3.0:1787827778.754085:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827778.754089:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827778.754094:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827778.754096:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827778.754109:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790333824/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827778.754119:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827778.754122:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827778.754125:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827778.754129:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4415 00000100:00080000:3.0:1787827778.754134:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827778.754138:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827778.754143:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827778.754171:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827778/real 1787827778] req@ffff9e680a2d5a40 x1874668790333440/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827778.754173:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790333952/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827778.754215:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827778.754219:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827778.754221:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827778.754226:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790333440/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827778.754229:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4415 00000100:00080000:3.0:1787827778.754236:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827778.754244:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787827778.754247:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787827778.754251:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787827778.754255:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827778.754262:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790333440:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787827778.754270:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790334080/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827778.754280:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827778.754284:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827778.754286:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827778.754291:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790333568:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827778.754295:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4415 00000100:00080000:3.0:1787827778.754303:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827778.754308:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827778.754314:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827778.754327:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790334208/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827778.754331:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827778.754345:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827778.754349:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827778.754351:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827778.754355:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827778/real 1787827778] req@ffff9e680a2d4780 x1874668790333568/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827778.754359:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4415 00000100:00080000:3.0:1787827778.754367:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827778.754375:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827778.754382:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790333568/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787827778.754382:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787827778.754399:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787827778.754402:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790334336/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827778.754407:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827778.754416:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790333568:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827778.754422:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827778.754426:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787827778.754428:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790333696:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827778.754429:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827778.754439:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4415 00000100:00080000:3.0:1787827778.754444:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827778.754449:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827778.754454:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827778.754462:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827778.754468:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790334464/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827778.754480:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827778/real 1787827778] req@ffff9e680a2d5e00 x1874668790333696/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827778.754496:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790333696/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827778.754505:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827778.754510:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827778.754514:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790333696:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827778.754522:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790333824:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827778.754544:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827778.754556:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827778/real 1787827778] req@ffff9e680a2d61c0 x1874668790333824/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827778.754569:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790333824/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827778.754578:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827778.754582:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827778.754586:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790333824:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827778.754592:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790333952:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827778.754611:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827778.754621:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827778/real 1787827778] req@ffff9e680a2d4000 x1874668790333952/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827778.754632:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790333952/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827778.754641:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827778.754644:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827778.754647:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790333952:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827778.754653:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790334080:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827778.754668:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827778.754679:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827778/real 1787827778] req@ffff9e680a2d52c0 x1874668790334080/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827778.754691:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790334080/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827778.754727:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827778.754731:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827778.754734:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790334080:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827778.754756:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790334208:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827778.754776:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827778.754786:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827778/real 1787827778] req@ffff9e680a2d6580 x1874668790334208/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827778.754798:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790334208/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827778.754805:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827778.754809:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827778.754811:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790334208:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827778.754817:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790334336:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827778.754845:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827778.754856:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827778/real 1787827778] req@ffff9e680a2d6d00 x1874668790334336/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827778.754867:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790334336/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827778.754881:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827778.754884:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827778.754887:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790334336:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827778.754892:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790334464:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827778.754909:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827778.754917:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827778/real 1787827778] req@ffff9e680a2d7c00 x1874668790334464/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827778.754928:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790334464/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827778.754935:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827778.754939:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827778.754942:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790334464:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827779.265841:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827779.457874:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827780.289811:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827780.481867:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827781.313792:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827781.505877:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827782.337750:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827782.529844:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827783.361785:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827783.553850:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827783.873780:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827783.873788:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827783.873791:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827783.873800:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4420 00000100:00080000:1.0:1787827783.873808:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827783.873814:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827783.873845:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790334592/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827783.873872:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827783.873876:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827783.873878:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827783.873883:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4420 00000100:00080000:1.0:1787827783.873888:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827783.873893:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827783.873904:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827783.873910:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790334592:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787827783.873927:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790334720/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827783.873937:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827783.873940:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827783.873942:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827783.873944:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4420 00000100:00080000:1.0:1787827783.873948:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827783.873950:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827783.873954:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827783.873962:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790334848/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827783.873968:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827783.873973:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1787827783.873973:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827783.873977:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827783.873982:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4420 00000100:00080000:1.0:1787827783.873986:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827783.873989:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827783.873992:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827783.874000:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790334976/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827783.874006:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827783.874008:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827783.874010:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827783.874012:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4420 00000100:00080000:1.0:1787827783.874015:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827783.874018:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827783.874022:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827783/real 1787827783] req@ffff9e6831aaa940 x1874668790334592/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787827783.874024:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827783.874037:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790335104/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827783.874045:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790334592/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827783.874046:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827783.874055:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827783.874057:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827783.874065:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4420 00000100:00080000:1.0:1787827783.874069:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827783.874069:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827783.874073:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787827783.874077:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:1.0:1787827783.874079:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827783.874083:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790334592:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787827783.874091:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790335232/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827783.874098:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827783.874100:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827783.874101:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827783.874107:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790334720:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827783.874128:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4420 00000100:00080000:1.0:1787827783.874132:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827783.874135:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827783.874140:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827783.874147:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827783.874152:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790335360/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827783.874161:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827783.874162:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827783/real 1787827783] req@ffff9e6831aa8f00 x1874668790334720/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827783.874168:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827783.874172:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827783.874180:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4420 00000100:00080000:1.0:1787827783.874183:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827783.874184:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790334720/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827783.874189:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827783.874196:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787827783.874200:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827783.874205:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827783.874207:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790335488/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827783.874211:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790334720:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827783.874223:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790334848:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827783.874225:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827783.874231:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827783.874232:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827783.874236:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4420 00000100:00080000:1.0:1787827783.874240:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827783.874243:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827783.874247:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827783.874248:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827783.874259:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790335616/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827783.874260:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827783/real 1787827783] req@ffff9e6831aaa1c0 x1874668790334848/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827783.874287:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790334848/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827783.874292:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827783.874295:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827783.874298:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790334848:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827783.874303:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790334976:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827783.874320:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827783.874330:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827783/real 1787827783] req@ffff9e6831aa8780 x1874668790334976/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827783.874342:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790334976/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827783.874352:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827783.874355:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827783.874358:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790334976:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827783.874364:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790335104:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827783.874380:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827783.874387:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827783/real 1787827783] req@ffff9e6831aa8000 x1874668790335104/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827783.874395:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790335104/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827783.874400:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827783.874402:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827783.874404:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790335104:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827783.874407:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790335232:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827783.874417:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827783.874422:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827783/real 1787827783] req@ffff9e6831aab480 x1874668790335232/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827783.874430:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790335232/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827783.874435:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827783.874437:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827783.874439:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790335232:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827783.874451:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790335360:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827783.874462:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827783.874467:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827783/real 1787827783] req@ffff9e6831aa9a40 x1874668790335360/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827783.874474:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790335360/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827783.874479:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827783.874481:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827783.874483:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790335360:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827783.874486:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790335488:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827783.874495:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827783.874500:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827783/real 1787827783] req@ffff9e6831aaad00 x1874668790335488/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827783.874508:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790335488/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827783.874517:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827783.874519:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827783.874521:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790335488:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827783.874524:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790335616:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827783.874533:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827783.874538:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827783/real 1787827783] req@ffff9e6831aa8b40 x1874668790335616/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827783.874544:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790335616/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827783.874549:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827783.874551:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827783.874553:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790335616:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827784.385769:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827784.577853:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827785.409792:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827785.601904:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827786.433813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827786.625921:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827787.457810:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827787.649934:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827788.481771:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827788.673873:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827788.993764:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827788.993774:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827788.993778:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827788.993788:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4425 00000100:00080000:1.0:1787827788.993798:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827788.993806:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827788.993842:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790335744/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827788.993872:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827788.993876:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827788.993879:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827788.993883:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4425 00000100:00080000:1.0:1787827788.993887:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827788.993892:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827788.993902:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827788.993915:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790335744:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787827788.993929:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790335872/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827788.993943:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827788.993947:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827788.993949:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827788.993954:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4425 00000100:00080000:1.0:1787827788.993958:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827788.993963:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827788.993969:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827788.993970:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827788.993988:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790336000/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827788.993999:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827788.994003:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827788.994005:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827788.994009:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827788/real 1787827788] req@ffff9e6831aa9680 x1874668790335744/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827788.994013:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4425 00000100:00080000:1.0:1787827788.994022:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827788.994029:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827788.994031:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790335744/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787827788.994034:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787827788.994045:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827788.994049:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827788.994052:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790336128/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827788.994056:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790335744:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827788.994069:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827788.994071:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827788.994073:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827788.994091:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4425 00000100:00080000:1.0:1787827788.994094:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827788.994097:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790335872:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827788.994101:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827788.994107:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827788.994118:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790336256/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827788.994126:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827788.994127:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827788.994131:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827788.994133:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827788.994137:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4425 00000100:00080000:1.0:1787827788.994140:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827788.994141:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827788/real 1787827788] req@ffff9e6831aabc00 x1874668790335872/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827788.994147:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827788.994154:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827788.994159:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790335872/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827788.994166:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790336384/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827788.994170:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827788.994176:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827788.994178:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790335872:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827788.994187:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790336000:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827788.994197:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827788.994200:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827788.994202:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827788.994206:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4425 00000800:00000100:2.0:1787827788.994211:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827788.994213:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827788.994218:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827788.994222:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827788.994223:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827788/real 1787827788] req@ffff9e6831aa9e00 x1874668790336000/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827788.994232:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790336512/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827788.994235:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790336000/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827788.994253:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827788.994256:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827788.994257:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827788.994260:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827788.994264:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787827788.994267:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4425 00000100:00100000:2.0:1787827788.994274:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790336000:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827788.994276:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827788.994284:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827788.994288:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827788.994289:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790336128:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827788.994297:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790336640/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827788.994304:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827788.994307:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827788.994309:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827788.994312:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4425 00000100:00080000:1.0:1787827788.994316:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827788.994319:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827788.994321:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827788.994326:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827788.994335:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827788/real 1787827788] req@ffff9e6831aaa580 x1874668790336128/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827788.994337:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790336768/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827788.994368:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790336128/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827788.994376:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827788.994379:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827788.994381:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790336128:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827788.994393:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790336256:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827788.994406:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827788.994412:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827788/real 1787827788] req@ffff9e6831aa9680 x1874668790336256/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827788.994419:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790336256/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827788.994424:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827788.994427:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827788.994428:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790336256:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827788.994432:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790336384:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827788.994441:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827788.994446:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827788/real 1787827788] req@ffff9e6831aa8b40 x1874668790336384/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827788.994454:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790336384/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827788.994461:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827788.994462:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827788.994464:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790336384:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827788.994475:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790336512:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827788.994486:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827788.994492:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827788/real 1787827788] req@ffff9e6831aaad00 x1874668790336512/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827788.994500:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790336512/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827788.994505:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827788.994507:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827788.994508:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790336512:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827788.994512:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790336640:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827788.994535:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827788.994547:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827788/real 1787827788] req@ffff9e6831aa9a40 x1874668790336640/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827788.994560:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790336640/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827788.994574:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827788.994576:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827788.994579:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790336640:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827788.994583:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790336768:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827788.994597:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827788.994604:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827788/real 1787827788] req@ffff9e6831aab480 x1874668790336768/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827788.994612:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790336768/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827788.994617:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827788.994619:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827788.994621:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790336768:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827789.505779:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827789.697846:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827790.529882:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827790.721869:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827791.553814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827791.745973:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827792.577788:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827792.769831:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827793.601837:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827793.794828:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827794.113912:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827794.113937:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827794.113943:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827794.113973:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4430 00000100:00080000:0.0:1787827794.113992:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827794.114005:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827794.114120:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790336896/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827794.114163:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827794.114170:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827794.114173:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827794.114181:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4430 00000100:00080000:0.0:1787827794.114190:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827794.114199:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827794.114216:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827794.114218:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790336896:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827794.114245:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790337024/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827794.114262:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827794.114267:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827794.114269:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827794.114274:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4430 00000100:00080000:0.0:1787827794.114279:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827794.114285:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827794.114292:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827794.114301:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827794.114309:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790337152/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827794.114323:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827794.114328:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827794.114331:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827794.114335:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4430 00000100:00080000:0.0:1787827794.114339:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827794.114344:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827794.114352:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827794.114358:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827794/real 1787827794] req@ffff9e6803e98b40 x1874668790336896/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827794.114370:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790337280/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827794.114385:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790336896/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827794.114427:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827794.114430:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827794.114434:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827794.114438:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827794.114445:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787827794.114450:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4430 00000100:00080000:0.0:1787827794.114460:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827794.114466:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827794.114468:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790336896:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1787827794.114472:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827794.114499:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790337408/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827794.114511:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790337024:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827794.114512:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827794.114543:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827794.114548:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827794.114556:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4430 00000100:00080000:0.0:1787827794.114561:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827794.114566:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827794.114570:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827794.114581:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827794.114589:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827794/real 1787827794] req@ffff9e6803e9bc00 x1874668790337024/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827794.114600:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790337536/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827794.114614:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790337024/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827794.114622:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827794.114633:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827794.114636:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827794.114641:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787827794.114645:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827794.114646:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790337024:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827794.114666:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4430 00000100:00100000:2.0:1787827794.114667:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790337152:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827794.114697:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827794.114706:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827794.114716:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827794.114733:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827794.114746:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827794/real 1787827794] req@ffff9e6803e992c0 x1874668790337152/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827794.114756:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790337152/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827794.114765:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827794.114768:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827794.114771:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790337152:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827794.114784:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790337280:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827794.114802:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827794.114809:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827794/real 1787827794] req@ffff9e6803e9ad00 x1874668790337280/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827794.114819:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790337280/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827794.114825:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827794.114832:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827794.114834:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790337664/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827794.114836:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790337280:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787827794.114866:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827794.114871:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787827794.114876:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790337408:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787827794.114877:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827794.114900:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4430 00000100:00080000:0.0:1787827794.114916:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827794.114926:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827794.114933:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827794.114935:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827794.114967:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827794/real 1787827794] req@ffff9e6803e98780 x1874668790337408/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827794.114977:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790337792/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827794.114988:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790337408/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827794.115005:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827794.115021:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827794.115024:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827794.115038:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4430 00000100:00080000:2.0:1787827794.115062:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827794.115068:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827794.115090:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787827794.115095:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827794.115104:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790337408:192.168.203.109@tcp:8:kworker.0 00000008:00100000:0.0:1787827794.115105:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827794.115127:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790337536:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827794.115134:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790337920/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827794.115179:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827794.115199:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827794/real 1787827794] req@ffff9e6803e9b480 x1874668790337536/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827794.115212:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790337536/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827794.115220:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827794.115224:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827794.115229:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790337536:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827794.115239:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790337664:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827794.115254:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827794.115269:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827794/real 1787827794] req@ffff9e6803e9a1c0 x1874668790337664/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827794.115279:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790337664/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827794.115285:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827794.115287:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827794.115289:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790337664:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827794.115303:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790337792:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827794.115316:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827794.115323:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827794/real 1787827794] req@ffff9e6803e9a580 x1874668790337792/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827794.115333:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790337792/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827794.115345:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827794.115347:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827794.115349:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790337792:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827794.115365:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790337920:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827794.115377:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827794.115384:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827794/real 1787827794] req@ffff9e6803e9b840 x1874668790337920/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827794.115393:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790337920/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827794.115399:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827794.115401:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827794.115404:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790337920:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827794.625778:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827794.817827:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827795.649802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827795.841912:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827796.673806:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827796.865830:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827797.697797:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827797.889908:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827798.721787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827798.913982:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827799.233837:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827799.233848:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827799.233854:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827799.233866:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4435 00000100:00080000:0.0:1787827799.233889:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827799.233898:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827799.233941:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790338048/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827799.233979:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827799.233984:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827799.233987:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827799.233993:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4435 00000100:00080000:0.0:1787827799.233999:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827799.234028:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827799.234042:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827799.234074:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790338176/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827799.234089:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827799.234094:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827799.234097:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827799.234103:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4435 00000100:00080000:0.0:1787827799.234109:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827799.234116:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827799.234123:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827799.234138:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790338304/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827799.234153:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827799.234177:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827799.234181:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827799.234188:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4435 00000100:00080000:0.0:1787827799.234196:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827799.234210:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790338048:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787827799.234218:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827799.234232:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827799.234255:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790338432/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827799.234279:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827799.234284:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827799.234287:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827799.234293:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4435 00000100:00080000:0.0:1787827799.234300:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787827799.234307:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827799.234311:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827799.234322:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827799.234341:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790338560/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827799.234354:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827799.234360:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827799.234362:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827799.234367:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4435 00000100:00080000:0.0:1787827799.234372:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827799.234377:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827799.234384:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827799.234398:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790338688/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827799.234412:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827799.234418:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827799.234422:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827799.234429:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4435 00000100:00080000:0.0:1787827799.234436:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827799.234443:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827799.234451:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827799.234471:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790338816/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827799.234488:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827799.234494:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827799.234497:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827799.234504:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4435 00000100:00080000:0.0:1787827799.234510:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827799.234516:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827799.234523:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827799.234555:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790338944/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827799.234566:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827799.234570:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827799.234572:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827799.234578:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4435 00000100:00080000:0.0:1787827799.234582:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827799.234585:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827799.234590:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827799.234605:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790339072/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827799.234643:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827799/real 1787827799] req@ffff9e6803e9a940 x1874668790338048/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827799.234693:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790338048/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827799.234708:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827799.234850:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827799.234861:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790338048:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787827799.234878:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790338176:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827799.234920:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827799.234940:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827799/real 1787827799] req@ffff9e6803e99680 x1874668790338176/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827799.234955:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790338176/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827799.234966:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827799.234970:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827799.234973:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790338176:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827799.234980:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790338304:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827799.235000:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827799.235050:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827799/real 1787827799] req@ffff9e6803e99e00 x1874668790338304/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827799.235086:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790338304/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827799.235102:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827799.235109:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827799.235115:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790338304:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827799.235185:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790338432:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827799.235218:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827799.235234:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827799/real 1787827799] req@ffff9e6803e98f00 x1874668790338432/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827799.235248:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790338432/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827799.235268:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827799.235272:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827799.235276:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790338432:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827799.235283:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790338560:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827799.235300:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827799.235310:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827799/real 1787827799] req@ffff9e6803e9b840 x1874668790338560/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827799.235323:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790338560/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827799.235358:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827799.235361:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827799.235365:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790338560:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827799.235370:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790338688:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827799.235389:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827799.235399:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827799/real 1787827799] req@ffff9e6803e9a580 x1874668790338688/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827799.235410:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790338688/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827799.235421:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827799.235424:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827799.235427:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790338688:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827799.235433:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790338816:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827799.235449:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827799.235478:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827799/real 1787827799] req@ffff9e6803e9a1c0 x1874668790338816/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827799.235491:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790338816/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827799.235500:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827799.235503:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827799.235507:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790338816:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827799.235514:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790338944:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827799.235533:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827799.235542:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827799/real 1787827799] req@ffff9e6803e9b480 x1874668790338944/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827799.235555:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790338944/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827799.235564:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827799.235567:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827799.235571:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790338944:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827799.235577:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790339072:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827799.235593:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827799.235603:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827799/real 1787827799] req@ffff9e6803e98780 x1874668790339072/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827799.235616:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790339072/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827799.235624:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827799.235627:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827799.235631:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790339072:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827799.745761:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827799.937831:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827800.769782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827800.961881:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827801.793860:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827801.985908:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827802.817787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827803.009854:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827803.841771:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827804.033871:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827804.353821:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827804.353828:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827804.353832:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827804.353840:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4441 00000100:00080000:1.0:1787827804.353849:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827804.353854:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827804.353888:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790339200/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827804.353912:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827804.353916:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827804.353918:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827804.353922:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4441 00000100:00080000:1.0:1787827804.353926:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827804.353930:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827804.353940:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827804.353956:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790339328/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827804.353963:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790339200:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827804.353990:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827804.353994:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827804.353996:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827804.354001:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4441 00000100:00080000:1.0:1787827804.354005:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827804.354009:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827804.354015:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827804.354028:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790339456/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827804.354036:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787827804.354042:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827804.354043:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827804.354047:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827804.354053:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4441 00000100:00080000:1.0:1787827804.354058:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827804.354062:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827804.354067:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827804.354079:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827804/real 1787827804] req@ffff9e6831aa8000 x1874668790339200/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827804.354082:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790339584/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827804.354097:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827804.354107:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827804.354109:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827804.354114:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790339200/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827804.354116:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4441 00000100:00080000:1.0:1787827804.354126:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827804.354129:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827804.354134:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:3.0:1787827804.354137:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827804.354144:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790339200:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787827804.354146:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827804.354160:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790339712/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827804.354166:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790339328:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827804.354174:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827804.354177:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827804.354179:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827804.354184:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4441 00000100:00080000:1.0:1787827804.354188:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787827804.354192:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827804.354196:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827804.354200:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827804.354205:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827804/real 1787827804] req@ffff9e6831aa8780 x1874668790339328/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827804.354211:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790339840/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827804.354225:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827804.354225:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790339328/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827804.354233:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827804.354236:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827804.354243:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4441 00000100:00080000:3.0:1787827804.354245:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827804.354246:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827804.354251:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787827804.354253:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827804.354257:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790339328:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787827804.354259:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827804.354265:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790339456:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827804.354271:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790339968/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827804.354283:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827804.354284:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827804.354289:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827804.354291:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827804.354295:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4441 00000100:00100000:3.0:1787827804.354298:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827804/real 1787827804] req@ffff9e6831aaa1c0 x1874668790339456/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827804.354300:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827804.354307:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827804.354314:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827804.354318:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790339456/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827804.354325:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790340096/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827804.354342:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827804.354344:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827804.354350:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827804.354353:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827804.354353:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787827804.354359:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4441 00000100:00100000:3.0:1787827804.354363:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790339456:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827804.354364:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827804.354370:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827804.354375:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827804.354378:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790339584:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827804.354383:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790340224/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827804.354400:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827804.354411:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827804/real 1787827804] req@ffff9e6831aa8f00 x1874668790339584/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827804.354419:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790339584/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827804.354425:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827804.354427:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827804.354429:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790339584:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827804.354441:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790339712:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827804.354452:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827804.354457:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827804/real 1787827804] req@ffff9e6831aaa940 x1874668790339712/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827804.354464:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790339712/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827804.354470:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827804.354472:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827804.354474:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790339712:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827804.354477:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790339840:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827804.354486:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827804.354491:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827804/real 1787827804] req@ffff9e6831aa8000 x1874668790339840/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827804.354498:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790339840/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827804.354504:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827804.354506:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827804.354508:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790339840:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827804.354511:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790339968:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827804.354520:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827804.354525:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827804/real 1787827804] req@ffff9e6831aab480 x1874668790339968/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827804.354532:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790339968/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827804.354537:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827804.354538:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827804.354541:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790339968:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827804.354544:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790340096:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827804.354553:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827804.354559:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827804/real 1787827804] req@ffff9e6831aa9a40 x1874668790340096/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827804.354565:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790340096/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827804.354570:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827804.354571:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827804.354573:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790340096:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827804.354576:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790340224:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827804.354586:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827804.354591:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827804/real 1787827804] req@ffff9e6831aaad00 x1874668790340224/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827804.354598:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790340224/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827804.354603:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827804.354604:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827804.354606:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790340224:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827804.865773:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827805.058057:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827805.889816:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827806.081996:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827806.913823:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827807.105921:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827807.937853:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827808.129904:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827808.961851:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827809.154051:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827809.473857:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827809.473871:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827809.473876:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827809.473889:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4446 00000100:00080000:0.0:1787827809.473905:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827809.473917:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827809.473994:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790340352/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827809.474053:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827809.474059:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827809.474061:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827809.474067:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4446 00000100:00080000:0.0:1787827809.474074:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827809.474080:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827809.474094:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827809.474115:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790340480/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827809.474131:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827809.474136:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790340352:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787827809.474137:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827809.474144:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827809.474159:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4446 00000100:00080000:0.0:1787827809.474163:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827809.474168:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827809.474174:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827809.474190:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790340608/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827809.474202:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827809.474207:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827809.474210:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827809.474214:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4446 00000100:00080000:0.0:1787827809.474231:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787827809.474239:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827809.474243:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827809.474252:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827809.474269:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790340736/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827809.474304:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827809.474304:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827809/real 1787827809] req@ffff9e6803e9ad00 x1874668790340352/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827809.474324:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827809.474335:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827809.474350:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790340352/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827809.474352:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4446 00000100:00080000:0.0:1787827809.474368:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827809.474379:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827809.474384:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:3.0:1787827809.474393:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:0.0:1787827809.474397:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827809.474402:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790340352:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827809.474427:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790340864/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827809.474442:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790340480:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827809.474448:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827809.474456:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827809.474459:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827809.474465:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4446 00000100:00080000:0.0:1787827809.474474:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787827809.474487:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827809.474498:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827809.474508:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827809.474513:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827809/real 1787827809] req@ffff9e6803e992c0 x1874668790340480/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827809.474531:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790340992/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827809.474544:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790340480/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827809.474555:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827809.474573:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827809.474574:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827809.474582:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827809.474594:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4446 00000100:00080000:3.0:1787827809.474594:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787827809.474605:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827809.474618:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827809.474618:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790340480:192.168.203.109@tcp:38:kworker.0 00000008:00100000:0.0:1787827809.474625:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827809.474649:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790341120/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827809.474652:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790340608:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827809.474681:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827809.474686:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827809.474689:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827809.474737:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4446 00000100:00080000:0.0:1787827809.474743:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827809.474748:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827809.474756:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827809.474774:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790341248/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827809.474789:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787827809.474799:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827809.474801:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827809.474809:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827809.474818:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4446 00000100:00080000:0.0:1787827809.474825:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827809.474837:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827809.474837:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827809/real 1787827809] req@ffff9e6803e9bc00 x1874668790340608/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787827809.474849:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827809.474873:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790341376/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827809.474880:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790340608/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827809.474922:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827809.474962:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827809.474972:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790340608:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827809.474984:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790340736:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827809.475017:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827809.475032:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827809/real 1787827809] req@ffff9e6803e98b40 x1874668790340736/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827809.475047:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790340736/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827809.475058:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827809.475063:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827809.475066:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790340736:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827809.475089:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790340864:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827809.475111:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827809.475122:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827809/real 1787827809] req@ffff9e6803e98780 x1874668790340864/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827809.475145:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790340864/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827809.475156:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827809.475159:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827809.475163:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790340864:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827809.475169:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790340992:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827809.475188:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827809.475199:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827809/real 1787827809] req@ffff9e6803e9b480 x1874668790340992/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827809.475212:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790340992/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827809.475223:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827809.475227:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827809.475230:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790340992:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827809.475237:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790341120:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827809.475255:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827809.475266:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827809/real 1787827809] req@ffff9e6803e9a1c0 x1874668790341120/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827809.475279:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790341120/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827809.475297:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827809.475301:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827809.475304:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790341120:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827809.475311:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790341248:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827809.475329:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827809.475339:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827809/real 1787827809] req@ffff9e6803e9a580 x1874668790341248/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827809.475352:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790341248/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827809.475363:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827809.475367:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827809.475370:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790341248:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827809.475377:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790341376:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827809.475394:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827809.475405:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827809/real 1787827809] req@ffff9e6803e9b840 x1874668790341376/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827809.475418:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790341376/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827809.475428:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827809.475432:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827809.475436:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790341376:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827809.985855:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827810.178079:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827811.009819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827811.202019:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827812.033764:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827812.225933:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827813.057768:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827813.249850:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827814.081833:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827814.273963:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827814.593817:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827814.593834:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827814.593842:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827814.593858:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4451 00000100:00080000:0.0:1787827814.593874:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827814.593888:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827814.593965:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790341504/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827814.594006:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827814.594011:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827814.594014:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827814.594018:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4451 00000100:00080000:0.0:1787827814.594024:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827814.594029:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827814.594042:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827814.594058:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790341632/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827814.594073:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827814.594077:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827814.594080:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827814.594085:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4451 00000100:00080000:0.0:1787827814.594091:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827814.594095:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790341504:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787827814.594100:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827814.594113:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827814.594127:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790341760/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827814.594141:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827814.594146:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827814.594149:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827814.594153:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4451 00000100:00080000:0.0:1787827814.594159:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827814.594162:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827814.594169:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827814.594184:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790341888/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827814.594191:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827814.594208:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827814.594214:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827814.594217:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827814.594225:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4451 00000100:00080000:0.0:1787827814.594229:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827814.594233:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827814.594239:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827814.594256:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790342016/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827814.594268:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827814.594275:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1787827814.594277:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827814/real 1787827814] req@ffff9e6803e98f00 x1874668790341504/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827814.594279:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827814.594287:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4451 00000100:00080000:0.0:1787827814.594295:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827814.594307:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827814.594319:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827814.594346:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790341504/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827814.594347:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790342144/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827814.594411:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827814.594413:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827814.594419:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827814.594427:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787827814.594428:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827814.594439:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790341504:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787827814.594441:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4451 00000100:00080000:0.0:1787827814.594453:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827814.594463:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790341632:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827814.594464:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827814.594478:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827814.594497:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790342272/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827814.594509:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787827814.594517:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827814.594520:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827814.594526:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827814.594534:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4451 00000100:00080000:0.0:1787827814.594540:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827814.594543:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827814/real 1787827814] req@ffff9e6803e99e00 x1874668790341632/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827814.594550:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827814.594562:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827814.594586:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790342400/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827814.594596:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790341632/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827814.594604:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827814.594618:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827814.594625:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827814.594637:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4451 00000100:00080000:0.0:1787827814.594645:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827814.594656:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827814.594669:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787827814.594680:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827814.594690:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827814.594691:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790342528/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827814.594733:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790341632:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827814.594776:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790341760:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827814.594826:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827814.594852:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827814/real 1787827814] req@ffff9e6803e99680 x1874668790341760/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827814.594878:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790341760/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827814.594891:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827814.594895:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827814.594927:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790341760:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827814.594936:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790341888:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827814.594967:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827814.594983:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827814/real 1787827814] req@ffff9e6803e9a940 x1874668790341888/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827814.594998:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790341888/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827814.595009:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827814.595013:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827814.595018:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790341888:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827814.595025:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790342016:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827814.595044:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827814.595054:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827814/real 1787827814] req@ffff9e6803e9b840 x1874668790342016/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827814.595067:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790342016/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827814.595078:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827814.595081:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827814.595084:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790342016:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827814.595091:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790342144:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827814.595109:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827814.595120:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827814/real 1787827814] req@ffff9e6803e9a580 x1874668790342144/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827814.595133:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790342144/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827814.595144:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827814.595148:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827814.595151:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790342144:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827814.595157:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790342272:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827814.595175:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827814.595187:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827814/real 1787827814] req@ffff9e6803e9a1c0 x1874668790342272/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827814.595201:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790342272/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827814.595210:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827814.595213:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827814.595217:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790342272:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827814.595223:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790342400:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827814.595241:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827814.595252:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827814/real 1787827814] req@ffff9e6803e9b480 x1874668790342400/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827814.595265:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790342400/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827814.595275:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827814.595278:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827814.595281:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790342400:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827814.595305:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790342528:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827814.595325:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827814.595335:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827814/real 1787827814] req@ffff9e6803e98780 x1874668790342528/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827814.595349:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790342528/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827814.595360:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827814.595363:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827814.595367:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790342528:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827815.105797:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827815.297877:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827816.129855:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827816.322016:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827817.153791:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827817.345982:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827818.177934:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827818.369994:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827819.201767:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827819.393946:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827819.713831:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827819.713846:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827819.713852:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827819.713914:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4456 00000100:00080000:0.0:1787827819.713932:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827819.713947:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827819.714018:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790342656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827819.714079:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827819.714084:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827819.714087:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827819.714092:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4456 00000100:00080000:0.0:1787827819.714099:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827819.714105:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827819.714118:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827819.714139:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790342784/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827819.714155:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827819.714160:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827819.714167:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827819.714169:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790342656:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787827819.714176:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4456 00000100:00080000:0.0:1787827819.714187:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827819.714193:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827819.714200:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827819.714221:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790342912/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827819.714234:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827819.714239:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827819.714243:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827819.714247:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4456 00000100:00080000:0.0:1787827819.714264:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787827819.714273:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827819.714284:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827819.714294:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827819.714343:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790343040/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827819.714362:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827819.714370:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787827819.714385:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827819/real 1787827819] req@ffff9e6803e98b40 x1874668790342656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827819.714397:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827819.714412:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4456 00000100:00080000:0.0:1787827819.714421:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827819.714423:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790342656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827819.714438:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827819.714452:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787827819.714479:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827819.714495:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827819.714498:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790343168/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827819.714516:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790342656:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787827819.714522:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827819.714545:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827819.714553:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827819.714563:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4456 00000100:00080000:0.0:1787827819.714603:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827819.714604:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790342784:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827819.714620:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827819.714633:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827819.714659:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790343296/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827819.714672:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827819.714701:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827819.714708:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787827819.714734:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827819/real 1787827819] req@ffff9e6803e9bc00 x1874668790342784/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827819.714749:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827819.714757:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4456 00000100:00080000:0.0:1787827819.714765:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827819.714771:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827819.714780:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827819.714796:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790343424/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827819.714819:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827819.714833:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827819.714841:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827819.714851:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4456 00000100:00080000:0.0:1787827819.714857:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827819.714911:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827819.714920:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790342784/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787827819.714924:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787827819.714950:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827819.714960:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827819.714973:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790343552/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827819.714975:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790342784:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827819.714995:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790342912:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827819.715018:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827819.715024:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827819.715027:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827819.715035:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4456 00000100:00080000:0.0:1787827819.715062:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787827819.715066:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827819.715075:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827819.715085:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827819.715098:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827819/real 1787827819] req@ffff9e6803e992c0 x1874668790342912/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827819.715110:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790343680/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827819.715134:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790342912/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827819.715149:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827819.715155:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827819.715161:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790342912:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827819.715171:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790343040:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827819.715196:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827819.715208:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827819/real 1787827819] req@ffff9e6803e9ad00 x1874668790343040/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827819.715222:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790343040/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827819.715233:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827819.715237:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827819.715242:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790343040:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827819.715248:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790343168:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827819.715267:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827819.715289:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827819/real 1787827819] req@ffff9e6803e98780 x1874668790343168/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827819.715302:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790343168/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827819.715313:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827819.715316:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827819.715320:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790343168:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827819.715368:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790343296:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827819.715401:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827819.715412:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827819/real 1787827819] req@ffff9e6803e9b480 x1874668790343296/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827819.715445:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790343296/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827819.715456:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827819.715460:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827819.715465:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790343296:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827819.715472:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790343424:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827819.715493:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827819.715504:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827819/real 1787827819] req@ffff9e6803e9a1c0 x1874668790343424/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827819.715517:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790343424/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827819.715542:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827819.715548:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827819.715554:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790343424:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827819.715566:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790343552:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827819.715618:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827819.715639:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827819/real 1787827819] req@ffff9e6803e9a580 x1874668790343552/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827819.715682:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790343552/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827819.715695:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827819.715700:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827819.715704:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790343552:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827819.715712:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790343680:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827819.715740:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827819.715755:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827819/real 1787827819] req@ffff9e6803e9b840 x1874668790343680/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827819.715769:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790343680/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827819.715820:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827819.715825:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827819.715829:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790343680:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827820.225783:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827820.417971:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827821.249839:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827821.442046:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827822.273875:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827822.466015:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827823.297863:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827823.490074:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827824.321810:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827824.513990:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827824.833783:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827824.833794:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827824.833800:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827824.833812:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4461 00000100:00080000:0.0:1787827824.833824:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827824.833855:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827824.833912:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790343808/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827824.833949:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827824.833953:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827824.833956:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827824.833977:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4461 00000100:00080000:0.0:1787827824.833982:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827824.833987:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827824.833998:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827824.834014:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790343936/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827824.834036:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827824.834041:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827824.834043:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827824.834048:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4461 00000100:00080000:0.0:1787827824.834051:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827824.834059:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827824.834065:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790343808:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1787827824.834067:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827824.834083:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790344064/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827824.834106:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827824.834110:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827824.834112:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827824.834117:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4461 00000100:00080000:0.0:1787827824.834121:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827824.834146:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827824.834153:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827824.834168:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790344192/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827824.834181:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827824.834189:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1787827824.834191:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827824.834193:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827824.834204:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4461 00000100:00080000:0.0:1787827824.834211:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827824.834217:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827824.834224:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827824.834238:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790344320/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827824.834249:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827824.834254:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827824.834258:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827824.834267:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4461 00000100:00100000:3.0:1787827824.834276:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827824/real 1787827824] req@ffff9e6803e9a940 x1874668790343808/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827824.834278:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827824.834297:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827824.834311:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827824.834314:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790343808/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827824.834334:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787827824.834343:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790344448/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827824.834345:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827824.834357:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790343808:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787827824.834379:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790343936:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827824.834385:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827824.834391:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827824.834394:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827824.834402:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4461 00000100:00080000:0.0:1787827824.834411:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827824.834417:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:3.0:1787827824.834425:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787827824.834432:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827824.834448:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827824/real 1787827824] req@ffff9e6803e99680 x1874668790343936/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827824.834458:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790344576/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827824.834501:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790343936/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827824.834518:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827824.834519:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827824.834527:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827824.834532:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827824.834533:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787827824.834540:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4461 00000100:00080000:0.0:1787827824.834547:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827824.834552:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827824.834557:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827824.834564:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790343936:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827824.834580:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790344704/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827824.834611:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790344064:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827824.834615:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827824.834636:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827824.834641:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827824.834651:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4461 00000800:00000100:3.0:1787827824.834653:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827824.834667:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827824.834677:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827824.834682:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827824/real 1787827824] req@ffff9e6803e99e00 x1874668790344064/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787827824.834727:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827824.834747:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790344832/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827824.834804:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790344064/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827824.834816:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827824.834820:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827824.834824:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790344064:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827824.834860:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790344192:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827824.834890:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827824.834905:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827824/real 1787827824] req@ffff9e6803e98f00 x1874668790344192/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827824.834919:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790344192/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827824.834929:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827824.834932:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827824.834936:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790344192:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827824.834942:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790344320:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827824.834959:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827824.834969:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827824/real 1787827824] req@ffff9e6803e9b840 x1874668790344320/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827824.834981:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790344320/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827824.834996:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827824.834999:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827824.835003:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790344320:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827824.835009:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790344448:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827824.835037:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827824.835056:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827824/real 1787827824] req@ffff9e6803e9a580 x1874668790344448/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827824.835079:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790344448/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827824.835111:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827824.835115:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827824.835119:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790344448:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827824.835127:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790344576:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827824.835153:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827824.835166:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827824/real 1787827824] req@ffff9e6803e9a1c0 x1874668790344576/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827824.835187:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790344576/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827824.835196:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827824.835199:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827824.835203:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790344576:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827824.835224:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790344704:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827824.835255:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827824.835264:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827824/real 1787827824] req@ffff9e6803e9b480 x1874668790344704/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827824.835277:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790344704/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827824.835286:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827824.835289:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827824.835293:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790344704:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827824.835301:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790344832:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827824.835332:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827824.835350:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827824/real 1787827824] req@ffff9e6803e98780 x1874668790344832/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827824.835373:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790344832/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827824.835389:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827824.835394:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827824.835400:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790344832:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827825.345924:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827825.538019:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827826.369801:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827826.561874:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827827.393825:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827827.586001:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827828.417779:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827828.609918:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827829.441824:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827829.633988:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827829.953914:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827829.953927:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827829.953933:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827829.953945:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4466 00000100:00080000:1.0:1787827829.953959:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827829.953968:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827829.954028:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790344960/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827829.954082:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827829.954087:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827829.954089:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827829.954094:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4466 00000100:00080000:1.0:1787827829.954098:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827829.954103:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827829.954115:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827829.954142:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790345088/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827829.954155:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790344960:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827829.954163:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827829.954187:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827829.954189:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827829.954195:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4466 00000100:00080000:1.0:1787827829.954199:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827829.954203:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827829.954210:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827829.954226:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790345216/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827829.954238:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827829.954242:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827829.954244:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827829.954248:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4466 00000100:00080000:1.0:1787827829.954255:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827829.954259:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827829.954264:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787827829.954274:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827829.954280:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790345344/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827829.954292:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827829.954297:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827829.954299:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827829.954304:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4466 00000100:00080000:1.0:1787827829.954308:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827829.954312:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827829.954317:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827829.954343:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790345472/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827829.954344:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827829/real 1787827829] req@ffff9e6831aa8b40 x1874668790344960/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827829.954389:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827829.954393:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827829.954407:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827829.954416:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4466 00000100:00100000:3.0:1787827829.954417:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790344960/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827829.954420:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827829.954430:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827829.954442:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787827829.954451:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787827829.954461:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790345600/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827829.954462:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827829.954474:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790344960:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827829.954509:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827829.954514:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827829.954517:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827829.954523:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4466 00000100:00080000:1.0:1787827829.954527:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827829.954532:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827829.954539:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827829.954540:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790345088:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827829.954559:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790345728/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827829.954575:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827829.954579:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827829.954582:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827829.954591:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4466 00000800:00000100:3.0:1787827829.954593:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827829.954596:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827829.954603:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827829.954610:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827829.954628:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790345856/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827829.954636:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827829/real 1787827829] req@ffff9e6831aa9680 x1874668790345088/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827829.954648:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827829.954664:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827829.954670:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827829.954683:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4466 00000100:00080000:1.0:1787827829.954689:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827829.954693:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827829.954698:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827829.954708:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790345984/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827829.954748:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790345088/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827829.954764:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827829.954771:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827829.954776:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790345088:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827829.954787:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790345216:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827829.954872:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827829.954916:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827829/real 1787827829] req@ffff9e6831aaa580 x1874668790345216/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827829.954932:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790345216/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827829.954943:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827829.954947:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827829.954951:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790345216:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827829.954960:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790345344:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827829.954991:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827829.955010:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827829/real 1787827829] req@ffff9e6831aa9e00 x1874668790345344/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827829.955050:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790345344/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827829.955066:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827829.955073:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827829.955079:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790345344:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827829.955091:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790345472:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827829.955127:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827829.955142:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827829/real 1787827829] req@ffff9e6831aabc00 x1874668790345472/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827829.955179:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790345472/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827829.955190:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827829.955195:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827829.955201:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790345472:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827829.955211:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790345600:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827829.955246:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827829.955266:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827829/real 1787827829] req@ffff9e6831aaad00 x1874668790345600/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827829.955286:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790345600/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827829.955297:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827829.955300:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827829.955304:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790345600:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827829.955328:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790345728:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827829.955353:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827829.955365:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827829/real 1787827829] req@ffff9e6831aa9a40 x1874668790345728/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827829.955378:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790345728/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827829.955386:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827829.955390:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827829.955393:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790345728:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827829.955410:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790345856:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827829.955425:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827829.955434:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827829/real 1787827829] req@ffff9e6831aab480 x1874668790345856/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827829.955446:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790345856/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827829.955462:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827829.955465:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827829.955468:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790345856:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827829.955473:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790345984:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827829.955508:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827829.955519:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827829/real 1787827829] req@ffff9e6831aa8000 x1874668790345984/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827829.955531:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790345984/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827829.955540:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827829.955543:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827829.955547:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790345984:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827830.465770:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827830.657843:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827831.489843:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827831.682038:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827832.513838:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827832.705980:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827833.537837:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827833.729901:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827834.561790:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827834.753896:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827835.073830:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827835.073840:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827835.073844:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827835.073872:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4471 00000100:00080000:1.0:1787827835.073886:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827835.073895:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827835.073942:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790346112/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827835.073980:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827835.073985:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827835.073988:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827835.073994:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4471 00000100:00080000:1.0:1787827835.073998:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827835.074003:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827835.074015:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827835.074030:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790346240/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827835.074043:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827835.074073:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827835.074075:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827835.074080:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4471 00000100:00080000:1.0:1787827835.074083:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827835.074088:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827835.074092:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827835.074114:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790346112:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787827835.074115:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790346368/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827835.074139:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827835.074144:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827835.074147:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827835.074153:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4471 00000100:00080000:1.0:1787827835.074161:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827835.074167:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827835.074175:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827835.074190:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790346496/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827835.074200:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827835.074203:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827835.074205:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827835.074208:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4471 00000100:00080000:1.0:1787827835.074214:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827835.074220:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827835.074224:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790346240:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787827835.074230:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827835.074266:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790346624/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827835.074276:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790346368:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827835.074290:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827835.074294:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827835.074296:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827835.074303:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790346496:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787827835.074311:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4471 00000100:00080000:1.0:1787827835.074318:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827835.074323:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827835.074330:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827835.074336:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790346624:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827835.074348:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790346752/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827835.074361:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827835.074364:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827835.074383:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827835.074386:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790346752:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827835.074395:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4471 00000100:00080000:1.0:1787827835.074406:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827835.074413:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827835.074422:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827835.074447:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790346880/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827835.074484:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827835.074491:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827835.074495:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827835.074505:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4471 00000100:00080000:1.0:1787827835.074514:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827835.074525:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827835.074529:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790346880:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787827835.074542:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827835.074578:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790347008/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827835.074605:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827835.074614:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827835.074619:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827835.074629:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790347008:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787827835.074632:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4471 00000100:00080000:1.0:1787827835.074650:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827835.074660:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827835.074670:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827835.074747:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790347136/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827835.074917:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790347136:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827835.585766:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827835.777856:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827836.609788:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827836.801867:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827837.633837:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827837.825911:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787827838.146043:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:0.0:1787827838.146090:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787827838.146106:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827838.146166:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:0.0:1787827838.146196:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827838.146207:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827838.146225:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827838.146234:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:3.0:1787827838.146234:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827835/real 1787827838] req@ffff9e6831aabc00 x1874668790347136/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787827838.146248:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827838.146260:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787827838.146268:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:3.0:1787827838.146280:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790347136/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827838.146292:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827838.146298:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827838.146308:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790347136:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827838.146319:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827835/real 1787827838] req@ffff9e6831aaad00 x1874668790347008/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827838.146354:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790347008/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827838.146366:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827838.146371:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827838.146376:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790347008:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827838.146386:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827835/real 1787827838] req@ffff9e6831aa9a40 x1874668790346880/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827838.146402:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790346880/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827838.146414:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827838.146420:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827838.146441:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790346880:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827838.146449:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827835/real 1787827838] req@ffff9e6831aab480 x1874668790346752/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827838.146471:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790346752/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827838.146481:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827838.146485:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827838.146488:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790346752:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827838.146496:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827835/real 1787827838] req@ffff9e6831aa8000 x1874668790346624/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827838.146516:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790346624/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827838.146524:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827838.146528:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827838.146532:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790346624:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827838.146539:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827835/real 1787827838] req@ffff9e6831aaa1c0 x1874668790346368/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827838.146552:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790346368/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827838.146577:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827838.146581:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827838.146586:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790346368:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827838.146594:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827835/real 1787827838] req@ffff9e6831aa8780 x1874668790346496/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827838.146608:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790346496/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827838.146619:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827838.146624:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827838.146628:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790346496:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827838.146635:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827835/real 1787827838] req@ffff9e6831aaa940 x1874668790346112/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827838.146652:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790346112/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827838.146662:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827838.146667:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827838.146671:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790346112:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787827838.146679:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827835/real 1787827838] req@ffff9e6831aa8f00 x1874668790346240/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827838.146718:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790346240/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827838.146730:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827838.146747:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827838.146752:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790346240:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827838.657868:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827838.850004:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827839.681788:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827839.873874:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827840.193804:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827840.193815:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827840.193820:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827840.193831:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4476 00000100:00080000:1.0:1787827840.193842:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827840.193850:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827840.193889:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790347264/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827840.193917:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827840.193921:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827840.193922:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827840.193926:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4476 00000100:00080000:1.0:1787827840.193930:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827840.193935:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827840.193947:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827840.193966:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790347264:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787827840.193970:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790347392/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827840.193994:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827840.193999:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827840.194003:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827840.194009:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4476 00000100:00080000:1.0:1787827840.194017:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827840.194024:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827840.194031:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787827840.194045:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827840.194054:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790347520/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827840.194071:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827840.194076:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827840.194078:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827840.194084:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4476 00000100:00080000:1.0:1787827840.194091:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827840.194097:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827840.194105:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827840.194117:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827840/real 1787827840] req@ffff9e6831aa9e00 x1874668790347264/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827840.194140:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790347648/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827840.194151:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790347264/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827840.194164:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827840.194178:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827840.194183:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827840.194197:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4476 00000100:00080000:3.0:1787827840.194199:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827840.194204:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827840.194212:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787827840.194218:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827840.194227:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827840.194250:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790347264:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787827840.194251:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790347776/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827840.194274:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827840.194280:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827840.194283:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827840.194289:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4476 00000100:00080000:1.0:1787827840.194296:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827840.194298:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790347392:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827840.194308:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827840.194320:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827840.194342:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790347904/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827840.194356:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827840.194373:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827840.194378:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827840.194381:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827840.194389:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827840/real 1787827840] req@ffff9e6831aaa580 x1874668790347392/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827840.194390:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4476 00000100:00080000:1.0:1787827840.194401:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827840.194413:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827840.194425:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827840.194430:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790347392/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827840.194448:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790348032/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827840.194451:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827840.194466:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787827840.194477:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827840.194480:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790347392:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827840.194488:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827840.194495:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827840.194502:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790347520:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827840.194507:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4476 00000100:00080000:1.0:1787827840.194518:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827840.194525:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827840.194535:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787827840.194553:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827840.194557:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790348160/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827840.194575:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827840.194582:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827840/real 1787827840] req@ffff9e6831aa9680 x1874668790347520/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827840.194584:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827840.194590:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827840.194603:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4476 00000100:00080000:1.0:1787827840.194614:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827840.194622:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790347520/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827840.194624:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827840.194638:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787827840.194649:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827840.194661:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827840.194665:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790348288/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827840.194672:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790347520:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827840.194766:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790347648:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827840.194803:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827840.195641:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827840/real 1787827840] req@ffff9e6831aa8b40 x1874668790347648/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827840.195667:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790347648/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827840.195680:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827840.195685:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827840.195690:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790347648:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827840.195748:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790347776:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827840.195790:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827840.195812:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827840/real 1787827840] req@ffff9e6831aa8f00 x1874668790347776/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827840.195829:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790347776/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827840.195841:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827840.195846:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827840.195851:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790347776:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827840.195885:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790347904:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827840.195919:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827840.195953:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827840/real 1787827840] req@ffff9e6831aaa940 x1874668790347904/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827840.195973:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790347904/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827840.195987:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827840.195992:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827840.195997:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790347904:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827840.196006:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790348032:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827840.196038:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827840.196073:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827840/real 1787827840] req@ffff9e6831aa8780 x1874668790348032/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827840.196091:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790348032/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827840.196103:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827840.196108:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827840.196112:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790348032:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827840.196121:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790348160:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827840.196151:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827840.196168:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827840/real 1787827840] req@ffff9e6831aaa1c0 x1874668790348160/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827840.196185:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790348160/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827840.196230:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827840.196235:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827840.196241:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790348160:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827840.196250:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790348288:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827840.196281:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827840.196298:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827840/real 1787827840] req@ffff9e6831aa8000 x1874668790348288/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827840.196313:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790348288/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827840.196324:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827840.196328:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827840.196332:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790348288:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827840.705811:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827840.897863:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827841.729817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827841.921929:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827842.753769:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827842.945842:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827843.777804:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827843.969852:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827844.801768:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827844.993881:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827845.313789:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827845.313798:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827845.313803:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827845.313812:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4482 00000100:00080000:1.0:1787827845.313822:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827845.313829:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827845.313864:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790348416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827845.313891:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827845.313895:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827845.313897:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827845.313901:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4482 00000100:00080000:1.0:1787827845.313905:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827845.313908:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827845.313918:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827845.313943:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790348544/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827845.313952:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790348416:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827845.313960:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827845.313964:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827845.313966:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827845.313972:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4482 00000100:00080000:1.0:1787827845.313976:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827845.313980:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827845.313985:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827845.313996:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790348672/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827845.314007:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787827845.314010:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827845.314013:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827845.314017:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827845.314032:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4482 00000100:00080000:1.0:1787827845.314037:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827845.314041:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827845.314046:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827845.314061:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827845/real 1787827845] req@ffff9e6831aab480 x1874668790348416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827845.314062:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790348800/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827845.314097:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827845.314101:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827845.314103:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827845.314106:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790348416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827845.314110:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4482 00000100:00080000:1.0:1787827845.314117:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827845.314120:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827845.314123:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:3.0:1787827845.314127:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:1.0:1787827845.314131:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827845.314131:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790348416:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787827845.314147:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790348928/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827845.314159:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827845.314161:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790348544:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827845.314166:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827845.314170:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827845.314175:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4482 00000100:00080000:1.0:1787827845.314178:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827845.314183:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827845.314188:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827845.314198:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790349056/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827845.314204:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827845.314214:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827845.314218:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827845.314220:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827845.314223:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827845/real 1787827845] req@ffff9e6831aa9a40 x1874668790348544/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827845.314226:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4482 00000100:00080000:1.0:1787827845.314233:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827845.314240:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827845.314245:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827845.314250:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790348544/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827845.314257:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790349184/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827845.314270:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827845.314275:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827845.314278:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787827845.314281:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787827845.314284:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790348544:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827845.314286:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827845.314294:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4482 00000100:00100000:3.0:1787827845.314296:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790348672:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827845.314299:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827845.314305:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827845.314310:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827845.314320:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790349312/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827845.314331:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827845.314332:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827845.314338:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827845.314341:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827845.314346:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4482 00000100:00080000:1.0:1787827845.314349:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827845.314351:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827845/real 1787827845] req@ffff9e6831aaad00 x1874668790348672/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827845.314355:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827845.314363:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827845.314377:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790348672/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827845.314379:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790349440/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827845.314435:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827845.314442:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827845.314446:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790348672:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827845.314454:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790348800:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827845.314473:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827845.314482:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827845/real 1787827845] req@ffff9e6831aabc00 x1874668790348800/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827845.314494:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790348800/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827845.314502:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827845.314505:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827845.314509:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790348800:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827845.314527:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790348928:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827845.314546:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827845.314554:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827845/real 1787827845] req@ffff9e6831aa8000 x1874668790348928/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827845.314561:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790348928/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827845.314567:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827845.314569:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827845.314572:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790348928:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827845.314576:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790349056:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827845.314586:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827845.314592:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827845/real 1787827845] req@ffff9e6831aaa1c0 x1874668790349056/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827845.314599:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790349056/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827845.314605:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827845.314607:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827845.314608:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790349056:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827845.314612:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790349184:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827845.314621:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827845.314627:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827845/real 1787827845] req@ffff9e6831aa8780 x1874668790349184/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827845.314633:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790349184/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827845.314638:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827845.314640:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827845.314642:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790349184:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827845.314645:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790349312:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827845.314654:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827845.314660:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827845/real 1787827845] req@ffff9e6831aaa940 x1874668790349312/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827845.314667:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790349312/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827845.314672:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827845.314674:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827845.314676:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790349312:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827845.314679:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790349440:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827845.314726:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827845.314734:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827845/real 1787827845] req@ffff9e6831aa8f00 x1874668790349440/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827845.314741:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790349440/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827845.314747:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827845.314749:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827845.314751:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790349440:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827845.825782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827846.017836:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827846.849798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827847.042198:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827847.873805:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827848.065909:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827848.897783:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827849.089901:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827849.921827:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827850.113906:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827850.433827:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827850.433841:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827850.433847:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827850.433860:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4487 00000100:00080000:1.0:1787827850.433876:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827850.433885:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827850.433955:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790349568/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827850.433990:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827850.433995:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827850.433997:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827850.434000:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4487 00000100:00080000:1.0:1787827850.434004:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827850.434008:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827850.434020:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827850.434033:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790349696/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827850.434045:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827850.434049:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827850.434051:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827850.434055:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4487 00000100:00080000:1.0:1787827850.434057:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827850.434063:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827850.434069:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827850.434078:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790349824/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827850.434093:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790349568:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827850.434098:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827850.434108:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827850.434110:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827850.434115:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4487 00000100:00080000:1.0:1787827850.434118:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827850.434133:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827850.434138:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827850.434152:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790349952/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827850.434161:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827850.434164:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827850.434167:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827850.434171:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4487 00000100:00080000:1.0:1787827850.434175:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827850.434179:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827850.434187:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827850.434210:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790350080/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827850.434218:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827850.434223:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827850.434225:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827850.434230:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4487 00000100:00080000:1.0:1787827850.434236:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787827850.434238:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827850.434255:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827850.434267:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827850.434292:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790350208/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827850.434305:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827850/real 1787827850] req@ffff9e6831aa8b40 x1874668790349568/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827850.434353:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827850.434359:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827850.434364:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827850.434374:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4487 00000100:00100000:3.0:1787827850.434375:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790349568/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827850.434387:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827850.434409:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827850.434424:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787827850.434425:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827850.434449:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827850.434466:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790350336/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827850.434466:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790349568:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827850.434500:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827850.434502:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790349696:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827850.434513:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827850.434519:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827850.434525:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4487 00000100:00080000:1.0:1787827850.434528:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827850.434534:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827850.434540:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787827850.434551:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827850.434559:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790350464/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827850.434574:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827850.434578:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827850/real 1787827850] req@ffff9e6831aa9680 x1874668790349696/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827850.434588:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827850.434602:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827850.434614:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790349696/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827850.434623:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4487 00000100:00080000:1.0:1787827850.434670:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827850.434685:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:3.0:1787827850.434729:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827850.434738:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827850.434744:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790349696:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787827850.434752:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827850.434783:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790350592/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827850.434790:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790349824:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827850.434849:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827850.434872:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827850/real 1787827850] req@ffff9e6831aaa580 x1874668790349824/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827850.434887:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790349824/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827850.434905:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827850.434909:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827850.434913:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790349824:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827850.434920:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790349952:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827850.434939:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827850.434971:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827850/real 1787827850] req@ffff9e6831aa9e00 x1874668790349952/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827850.434995:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790349952/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827850.435011:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827850.435018:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827850.435025:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790349952:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827850.435038:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790350080:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827850.435079:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827850.435096:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827850/real 1787827850] req@ffff9e6831aa8f00 x1874668790350080/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827850.435111:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790350080/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827850.435121:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827850.435125:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827850.435130:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790350080:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827850.435138:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790350208:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827850.435161:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827850.435172:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827850/real 1787827850] req@ffff9e6831aaa940 x1874668790350208/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827850.435186:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790350208/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827850.435215:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827850.435218:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827850.435222:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790350208:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827850.435228:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790350336:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827850.435249:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827850.435260:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827850/real 1787827850] req@ffff9e6831aa8780 x1874668790350336/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827850.435273:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790350336/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827850.435281:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827850.435284:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827850.435288:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790350336:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827850.435294:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790350464:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827850.435311:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827850.435340:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827850/real 1787827850] req@ffff9e6831aaa1c0 x1874668790350464/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827850.435354:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790350464/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827850.435364:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827850.435367:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827850.435371:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790350464:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827850.435397:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790350592:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827850.435420:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827850.435431:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827850/real 1787827850] req@ffff9e6831aa8000 x1874668790350592/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827850.435445:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790350592/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827850.435455:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827850.435458:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827850.435462:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790350592:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827850.945836:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827851.137933:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827851.969884:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827852.161901:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827852.993806:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827853.186034:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827854.017788:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827854.209914:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827855.041793:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827855.233915:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827855.553751:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827855.553759:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827855.553763:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827855.553770:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4492 00000100:00080000:1.0:1787827855.553779:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827855.553784:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827855.553817:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790350720/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827855.553840:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827855.553843:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827855.553848:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827855.553852:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4492 00000100:00080000:1.0:1787827855.553854:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827855.553857:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827855.553866:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827855.553876:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790350848/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827855.553885:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827855.553897:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790350720:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827855.553899:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827855.553903:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827855.553908:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4492 00000100:00080000:1.0:1787827855.553910:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827855.553912:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827855.553916:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827855.553924:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790350976/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827855.553930:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827855.553933:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827855.553934:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827855.553938:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4492 00000100:00080000:1.0:1787827855.553940:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827855.553942:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827855.553946:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827855.553958:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790351104/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827855.553959:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827855.553968:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827855.553971:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827855.553972:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827855.553976:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4492 00000100:00080000:1.0:1787827855.553978:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827855.553980:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827855.553983:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827855.553992:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790351232/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827855.553998:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827855/real 1787827855] req@ffff9e6831aabc00 x1874668790350720/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827855.554000:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827855.554008:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827855.554011:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827855.554019:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4492 00000100:00080000:1.0:1787827855.554025:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827855.554032:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827855.554036:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827855.554039:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790350720/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827855.554045:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790351360/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827855.554052:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827855.554055:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827855.554059:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787827855.554061:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827855.554064:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827855.554065:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790350720:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827855.554070:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4492 00000100:00080000:1.0:1787827855.554074:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827855.554077:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827855.554080:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827855.554086:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790351488/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827855.554091:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790350848:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827855.554097:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827855.554100:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827855.554102:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827855.554105:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4492 00000100:00080000:1.0:1787827855.554109:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827855.554112:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827855.554116:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787827855.554118:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827855.554129:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790351616/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827855.554133:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827855/real 1787827855] req@ffff9e6831aaad00 x1874668790350848/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827855.554149:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827855.554154:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827855.554157:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827855.554157:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790350848/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827855.554166:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4492 00000100:00080000:1.0:1787827855.554171:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827855.554176:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:3.0:1787827855.554176:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787827855.554184:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787827855.554184:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827855.554190:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790350848:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827855.554195:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790351744/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827855.554198:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790350976:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827855.554221:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827855.554231:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827855/real 1787827855] req@ffff9e6831aa9a40 x1874668790350976/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827855.554239:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790350976/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827855.554249:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827855.554251:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827855.554253:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790350976:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827855.554257:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790351104:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827855.554267:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827855.554272:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827855/real 1787827855] req@ffff9e6831aab480 x1874668790351104/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827855.554279:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790351104/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827855.554284:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827855.554286:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827855.554288:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790351104:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827855.554291:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790351232:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827855.554300:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827855.554306:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827855/real 1787827855] req@ffff9e6831aa8000 x1874668790351232/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827855.554312:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790351232/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827855.554317:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827855.554319:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827855.554322:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790351232:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827855.554325:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790351360:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827855.554334:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827855.554339:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827855/real 1787827855] req@ffff9e6831aaa1c0 x1874668790351360/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827855.554346:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790351360/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827855.554351:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827855.554353:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827855.554355:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790351360:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827855.554368:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790351488:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827855.554378:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827855.554383:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827855/real 1787827855] req@ffff9e6831aa8780 x1874668790351488/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827855.554390:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790351488/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827855.554395:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827855.554397:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827855.554398:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790351488:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827855.554402:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790351616:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827855.554410:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827855.554419:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827855/real 1787827855] req@ffff9e6831aaa940 x1874668790351616/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827855.554431:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790351616/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827855.554439:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827855.554443:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827855.554447:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790351616:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827855.554453:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790351744:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827855.554474:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827855.554483:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827855/real 1787827855] req@ffff9e6831aa8f00 x1874668790351744/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827855.554491:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790351744/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827855.554497:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827855.554499:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827855.554501:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790351744:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827856.065787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827856.257848:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827857.089775:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827857.281885:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827858.113782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827858.305906:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827859.137812:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827859.329906:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827860.161828:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827860.353826:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827860.673793:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827860.673804:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827860.673809:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827860.673819:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4497 00000100:00080000:2.0:1787827860.673830:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827860.673839:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827860.673889:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790351872/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827860.673918:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827860.673922:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827860.673925:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827860.673930:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4497 00000100:00080000:2.0:1787827860.673934:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827860.673938:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787827860.673948:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827860.673964:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790352000/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827860.673972:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790351872:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787827860.673993:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827860.673998:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827860.674000:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827860.674006:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4497 00000100:00080000:2.0:1787827860.674021:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827860.674027:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787827860.674034:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787827860.674034:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827860.674058:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790352128/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827860.674070:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827860.674075:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827860.674079:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827860.674079:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827860/real 1787827860] req@ffff9e6804423480 x1874668790351872/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827860.674092:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4497 00000100:00080000:2.0:1787827860.674099:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827860.674104:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790351872/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827860.674107:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787827860.674119:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787827860.674121:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827860.674129:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827860.674133:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790351872:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787827860.674140:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790352256/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827860.674155:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790352000:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787827860.674156:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827860.674164:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827860.674167:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827860.674174:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4497 00000800:00000100:3.0:1787827860.674181:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787827860.674184:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827860.674192:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827860.674195:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827860/real 1787827860] req@ffff9e68044230c0 x1874668790352000/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787827860.674203:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827860.674209:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790352000/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827860.674218:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787827860.674223:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790352384/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827860.674224:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827860.674230:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790352000:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827860.674242:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790352128:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787827860.674252:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827860.674258:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827860.674261:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787827860.674265:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787827860.674270:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4497 00000100:00100000:3.0:1787827860.674276:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827860/real 1787827860] req@ffff9e6804422580 x1874668790352128/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827860.674278:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827860.674289:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827860.674293:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790352128/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787827860.674299:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787827860.674317:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787827860.674318:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790352512/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827860.674324:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827860.674331:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790352128:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787827860.674344:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827860.674349:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787827860.674352:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790352256:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787827860.674353:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827860.674362:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4497 00000100:00080000:2.0:1787827860.674369:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787827860.674372:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787827860.674394:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827860.674402:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827860/real 1787827860] req@ffff9e6804420b40 x1874668790352256/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787827860.674403:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827860.674419:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790352256/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827860.674427:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827860.674429:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827860.674434:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790352256:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827860.674439:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790352640/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827860.674448:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790352384:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787827860.674469:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827860.674472:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827860.674474:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827860.674479:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4497 00000100:00080000:2.0:1787827860.674483:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787827860.674487:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787827860.674492:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787827860.674499:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827860.674512:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827860/real 1787827860] req@ffff9e6804423480 x1874668790352384/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827860.674513:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790352768/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827860.674555:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827860.674560:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787827860.674561:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790352384/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827860.674563:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827860.674573:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4497 00000100:00080000:2.0:1787827860.674595:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827860.674600:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827860.674601:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:3.0:1787827860.674611:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:2.0:1787827860.674612:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827860.674621:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790352384:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827860.674644:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790352896/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827860.674646:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790352512:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827860.674676:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827860.674687:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827860/real 1787827860] req@ffff9e68044221c0 x1874668790352512/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827860.674697:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790352512/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827860.674707:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827860.674737:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827860.674742:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790352512:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827860.674760:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790352640:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827860.674778:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827860.674787:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827860/real 1787827860] req@ffff9e6804423840 x1874668790352640/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827860.674796:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790352640/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827860.674802:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827860.674804:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827860.674806:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790352640:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827860.674810:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790352768:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827860.674821:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827860.674828:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827860/real 1787827860] req@ffff9e68044212c0 x1874668790352768/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827860.674835:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790352768/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827860.674841:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827860.674843:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827860.674845:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790352768:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827860.674848:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790352896:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827860.674858:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827860.674864:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827860/real 1787827860] req@ffff9e6804421680 x1874668790352896/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827860.674871:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790352896/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827860.674876:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827860.674878:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827860.674880:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790352896:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827861.185818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827861.377854:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827862.209773:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827862.401836:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827863.233815:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827863.425935:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827864.257774:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827864.449979:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827865.281806:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827865.473931:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827865.793799:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827865.793812:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827865.793818:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827865.793833:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4502 00000100:00080000:2.0:1787827865.793849:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827865.793858:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827865.793908:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790353024/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827865.793941:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827865.793945:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827865.793947:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827865.793952:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4502 00000100:00080000:2.0:1787827865.793957:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827865.793963:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787827865.793975:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827865.793990:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790353024:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787827865.793994:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790353152/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827865.794008:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827865.794012:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827865.794014:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827865.794019:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4502 00000100:00080000:2.0:1787827865.794024:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827865.794028:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787827865.794034:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827865.794046:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790353280/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827865.794056:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827865.794060:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827865.794063:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787827865.794066:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787827865.794071:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4502 00000100:00080000:2.0:1787827865.794077:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827865.794083:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787827865.794089:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827865.794102:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790353408/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827865.794113:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827865.794113:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827865/real 1787827865] req@ffff9e6804420780 x1874668790353024/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827865.794124:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827865.794128:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827865.794136:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4502 00000100:00080000:2.0:1787827865.794141:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827865.794142:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790353024/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827865.794152:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:3.0:1787827865.794155:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787827865.794160:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787827865.794162:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827865.794169:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790353024:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787827865.794183:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790353536/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827865.794196:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827865.794196:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790353152:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787827865.794206:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827865.794209:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827865.794214:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4502 00000100:00080000:2.0:1787827865.794220:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787827865.794223:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787827865.794229:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827865.794236:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827865/real 1787827865] req@ffff9e6804420f00 x1874668790353152/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787827865.794237:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827865.794252:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790353152/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827865.794257:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790353664/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827865.794264:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827865.794272:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787827865.794278:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827865.794278:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790353152:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787827865.794287:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787827865.794288:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790353280:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787827865.794292:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827865.794299:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4502 00000100:00080000:2.0:1787827865.794305:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787827865.794307:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787827865.794314:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827865.794319:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827865/real 1787827865] req@ffff9e6804422d00 x1874668790353280/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787827865.794323:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827865.794332:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790353280/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827865.794342:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790353792/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827865.794375:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827865.794378:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827865.794381:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787827865.794386:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787827865.794388:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790353280:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787827865.794390:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827865.794396:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790353408:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787827865.794401:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4502 00000100:00080000:2.0:1787827865.794408:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827865.794414:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:3.0:1787827865.794417:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:2.0:1787827865.794424:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827865.794429:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827865/real 1787827865] req@ffff9e6804421e00 x1874668790353408/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827865.794445:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790353408/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827865.794446:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790353920/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827865.794460:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827865.794466:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827865.794471:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790353408:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787827865.794472:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827865.794480:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827865.794483:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827865.794488:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4502 00000100:00100000:3.0:1787827865.794492:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790353536:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787827865.794495:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827865.794502:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787827865.794509:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787827865.794509:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827865.794519:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827865/real 1787827865] req@ffff9e6804422940 x1874668790353536/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827865.794528:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790354048/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827865.794533:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790353536/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827865.794557:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827865.794563:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827865.794569:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790353536:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827865.794575:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790353664:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827865.794589:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827865.794603:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827865/real 1787827865] req@ffff9e6804420780 x1874668790353664/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827865.794611:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790353664/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827865.794618:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827865.794620:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827865.794622:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790353664:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827865.794626:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790353792:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827865.794637:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827865.794643:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827865/real 1787827865] req@ffff9e6804421680 x1874668790353792/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827865.794651:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790353792/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827865.794656:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827865.794657:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827865.794659:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790353792:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827865.794663:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790353920:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827865.794672:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827865.794678:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827865/real 1787827865] req@ffff9e68044212c0 x1874668790353920/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827865.794686:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790353920/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827865.794691:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827865.794693:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827865.794714:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790353920:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827865.794725:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790354048:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827865.794738:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827865.794745:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827865/real 1787827865] req@ffff9e6804423840 x1874668790354048/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827865.794753:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790354048/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827865.794765:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827865.794768:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827865.794770:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790354048:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827866.305800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827866.497912:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827867.329796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827867.521988:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827868.353780:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827868.545896:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827869.377784:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827869.569843:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827870.401814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827870.593945:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827870.913874:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827870.913889:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827870.913895:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827870.913909:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4507 00000100:00080000:0.0:1787827870.913924:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827870.913934:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827870.913984:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790354176/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827870.914024:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827870.914029:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827870.914032:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827870.914038:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4507 00000100:00080000:0.0:1787827870.914044:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827870.914050:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827870.914066:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827870.914101:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790354176:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827870.914104:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790354304/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827870.914131:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827870.914137:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827870.914140:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827870.914147:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4507 00000100:00080000:0.0:1787827870.914153:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827870.914158:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827870.914166:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827870.914181:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790354432/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827870.914196:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787827870.914202:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827870.914209:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827870.914214:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827870.914222:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4507 00000100:00080000:0.0:1787827870.914242:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827870.914253:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827870.914262:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827870/real 1787827870] req@ffff9e6803e9ad00 x1874668790354176/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787827870.914266:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827870.914298:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790354176/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827870.914309:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790354560/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827870.914322:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827870.914341:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827870.914354:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790354176:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787827870.914355:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827870.914367:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827870.914372:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827870.914380:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4507 00000100:00080000:0.0:1787827870.914386:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827870.914393:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827870.914400:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790354304:192.168.203.109@tcp:38:kworker.0 00000008:00100000:0.0:1787827870.914405:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827870.914433:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790354688/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827870.914449:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827870.914455:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827870.914463:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827870.914466:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827870.914473:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827870/real 1787827870] req@ffff9e6803e992c0 x1874668790354304/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827870.914475:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4507 00000100:00080000:0.0:1787827870.914488:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827870.914502:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827870.914535:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827870.914536:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790354304/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827870.914573:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787827870.914575:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790354816/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827870.914587:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827870.914604:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790354304:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827870.914609:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827870.914618:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790354432:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827870.914635:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827870.914641:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827870.914650:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4507 00000100:00080000:0.0:1787827870.914658:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827870.914665:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787827870.914673:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787827870.914679:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827870.914740:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827870/real 1787827870] req@ffff9e6803e9bc00 x1874668790354432/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827870.914775:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790354944/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827870.914776:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790354432/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827870.914842:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827870.914848:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827870.914849:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827870.914855:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827870.914869:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787827870.914871:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4507 00000100:00080000:0.0:1787827870.914884:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827870.914892:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790354432:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827870.914894:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827870.914913:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827870.914923:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790354560:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827870.914935:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790355072/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827870.914951:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827870.914956:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827870.914959:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827870.914964:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4507 00000800:00000100:2.0:1787827870.914973:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827870.914975:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827870.914985:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827870.914992:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827870.915002:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827870/real 1787827870] req@ffff9e6803e98b40 x1874668790354560/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827870.915013:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790355200/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827870.915039:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790354560/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827870.915054:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827870.915059:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827870.915076:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790354560:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827870.915101:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790354688:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827870.915125:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827870.915138:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827870/real 1787827870] req@ffff9e6803e9ad00 x1874668790354688/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827870.915152:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790354688/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827870.915162:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827870.915166:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827870.915170:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790354688:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827870.915176:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790354816:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827870.915195:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827870.915206:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827870/real 1787827870] req@ffff9e6803e98780 x1874668790354816/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827870.915220:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790354816/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827870.915232:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827870.915235:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827870.915239:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790354816:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827870.915245:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790354944:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827870.915263:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827870.915274:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827870/real 1787827870] req@ffff9e6803e9b480 x1874668790354944/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827870.915288:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790354944/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827870.915298:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827870.915301:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827870.915304:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790354944:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827870.915311:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790355072:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827870.915329:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827870.915339:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827870/real 1787827870] req@ffff9e6803e9a1c0 x1874668790355072/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827870.915352:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790355072/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827870.915361:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827870.915365:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827870.915369:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790355072:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827870.915375:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790355200:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827870.915394:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827870.915405:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827870/real 1787827870] req@ffff9e6803e9a580 x1874668790355200/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827870.915418:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790355200/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827870.915428:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827870.915432:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827870.915436:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790355200:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827871.425876:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827871.617953:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827872.449808:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827872.641954:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827873.473902:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827873.665969:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787827874.497830:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827874.689878:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827875.521786:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827875.714041:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827876.033788:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827876.033796:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827876.033800:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827876.033808:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4512 00000100:00080000:2.0:1787827876.033816:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827876.033822:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827876.033855:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790355328/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827876.033879:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827876.033882:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827876.033883:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827876.033886:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4512 00000100:00080000:2.0:1787827876.033888:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827876.033890:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787827876.033898:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827876.033919:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790355456/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827876.033930:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790355328:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787827876.033934:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827876.033942:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827876.033944:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827876.033950:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4512 00000100:00080000:2.0:1787827876.033954:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827876.033959:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787827876.033966:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827876.033978:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790355584/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827876.033986:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787827876.033989:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787827876.033991:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827876.033994:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827876.033998:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4512 00000100:00080000:2.0:1787827876.034002:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827876.034005:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787827876.034009:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827876.034018:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790355712/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827876.034024:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827876.034028:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787827876.034029:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827876/real 1787827876] req@ffff9e68044221c0 x1874668790355328/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827876.034031:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827876.034037:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4512 00000100:00080000:2.0:1787827876.034043:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827876.034049:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787827876.034053:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827876.034057:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790355328/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827876.034065:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790355840/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827876.034069:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827876.034077:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827876.034078:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787827876.034085:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1787827876.034086:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790355328:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787827876.034089:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827876.034095:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4512 00000100:00080000:2.0:1787827876.034097:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827876.034100:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787827876.034102:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827876.034110:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790355968/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827876.034113:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790355456:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787827876.034127:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827876.034130:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827876.034132:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827876.034135:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4512 00000100:00080000:2.0:1787827876.034138:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787827876.034142:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787827876.034145:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787827876.034150:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827876.034158:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827876/real 1787827876] req@ffff9e6804423480 x1874668790355456/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827876.034162:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790356096/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827876.034183:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827876.034185:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790355456/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827876.034187:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827876.034189:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827876.034192:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4512 00000100:00080000:2.0:1787827876.034194:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827876.034197:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787827876.034203:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787827876.034210:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827876.034215:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827876.034217:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790355456:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827876.034222:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790356224/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827876.034225:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790355584:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787827876.034239:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827876.034243:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827876.034244:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827876.034249:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4512 00000800:00000100:3.0:1787827876.034251:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787827876.034254:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827876.034259:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787827876.034263:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827876.034266:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827876/real 1787827876] req@ffff9e6804420b40 x1874668790355584/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827876.034301:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790356352/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827876.034313:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790355584/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827876.034327:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827876.034331:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827876.034336:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790355584:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827876.034344:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790355712:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827876.034363:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827876.034381:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827876/real 1787827876] req@ffff9e6804422580 x1874668790355712/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827876.034390:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790355712/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827876.034396:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827876.034398:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827876.034401:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790355712:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827876.034405:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790355840:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827876.034415:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827876.034421:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827876/real 1787827876] req@ffff9e68044230c0 x1874668790355840/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827876.034429:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790355840/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827876.034448:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827876.034450:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827876.034452:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790355840:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827876.034467:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790355968:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827876.034534:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827876.034547:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827876/real 1787827876] req@ffff9e68044221c0 x1874668790355968/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827876.034559:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790355968/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827876.034567:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827876.034586:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827876.034589:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790355968:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827876.034594:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790356096:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827876.034611:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827876.034620:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827876/real 1787827876] req@ffff9e6804423840 x1874668790356096/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827876.034628:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790356096/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827876.034634:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827876.034651:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827876.034654:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790356096:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827876.034669:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790356224:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827876.034680:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827876.034686:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827876/real 1787827876] req@ffff9e68044212c0 x1874668790356224/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827876.034693:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790356224/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827876.034698:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827876.034699:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827876.034701:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790356224:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827876.034704:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790356352:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827876.034714:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827876.034719:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827876/real 1787827876] req@ffff9e6804421680 x1874668790356352/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827876.034747:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790356352/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827876.034763:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827876.034765:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827876.034767:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790356352:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827876.545800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827876.737966:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827877.569819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827877.761946:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827878.593850:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827878.785977:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827879.617812:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827879.809931:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827880.641780:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827880.833878:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827881.153810:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827881.153819:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827881.153822:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827881.153830:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4517 00000100:00080000:1.0:1787827881.153839:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827881.153845:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827881.153877:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790356480/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827881.153902:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827881.153905:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827881.153907:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827881.153909:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4517 00000100:00080000:1.0:1787827881.153911:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827881.153914:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827881.153921:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827881.153930:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790356608/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827881.153937:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827881.153940:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827881.153942:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827881.153944:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4517 00000100:00080000:1.0:1787827881.153946:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827881.153951:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827881.153952:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790356480:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787827881.153954:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827881.153964:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790356736/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827881.153970:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827881.153972:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827881.153973:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827881.153976:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4517 00000100:00080000:1.0:1787827881.153978:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827881.153981:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827881.153984:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827881.153991:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790356864/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827881.153997:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827881.154000:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827881.154001:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827881.154004:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4517 00000100:00080000:1.0:1787827881.154006:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827881.154008:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827881.154011:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787827881.154016:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827881.154023:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790356992/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827881.154030:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827881.154033:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827881.154034:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827881.154037:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4517 00000100:00080000:1.0:1787827881.154039:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827881.154041:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827881.154044:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827881.154051:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790357120/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827881.154057:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827881.154059:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827881.154062:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827881.154064:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827881/real 1787827881] req@ffff9e6831aa9e00 x1874668790356480/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827881.154067:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4517 00000100:00080000:1.0:1787827881.154071:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827881.154075:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827881.154079:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827881.154088:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790357248/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827881.154094:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827881.154098:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827881.154100:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827881.154100:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790356480/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827881.154107:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4517 00000100:00080000:1.0:1787827881.154111:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827881.154117:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827881.154122:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787827881.154127:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787827881.154130:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790357376/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827881.154135:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827881.154145:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790356480:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827881.154148:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827881.154154:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827881.154156:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827881.154159:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790356608:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827881.154164:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4517 00000100:00080000:1.0:1787827881.154170:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827881.154174:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827881.154179:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787827881.154192:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827881.154195:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790357504/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827881.154209:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827881/real 1787827881] req@ffff9e6831aaa580 x1874668790356608/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827881.154226:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790356608/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827881.154236:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827881.154239:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827881.154243:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790356608:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827881.154268:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790356736:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827881.154284:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827881.154292:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827881/real 1787827881] req@ffff9e6831aa9680 x1874668790356736/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827881.154300:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790356736/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827881.154306:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827881.154308:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827881.154310:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790356736:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827881.154313:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790356864:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827881.154323:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827881.154329:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827881/real 1787827881] req@ffff9e6831aa8b40 x1874668790356864/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827881.154336:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790356864/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827881.154341:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827881.154343:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827881.154345:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790356864:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827881.154348:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790356992:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827881.154364:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827881.154375:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827881/real 1787827881] req@ffff9e6831aa8f00 x1874668790356992/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827881.154387:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790356992/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827881.154396:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827881.154400:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827881.154404:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790356992:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827881.154411:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790357120:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827881.154429:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827881.154450:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827881/real 1787827881] req@ffff9e6831aaa940 x1874668790357120/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827881.154464:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790357120/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827881.154473:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827881.154476:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827881.154480:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790357120:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827881.154486:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790357248:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827881.154509:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827881.154522:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827881/real 1787827881] req@ffff9e6831aa8780 x1874668790357248/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827881.154535:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790357248/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827881.154544:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827881.154547:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827881.154551:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790357248:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827881.154558:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790357376:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827881.154576:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827881.154586:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827881/real 1787827881] req@ffff9e6831aaa1c0 x1874668790357376/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827881.154598:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790357376/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827881.154606:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827881.154609:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827881.154613:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790357376:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827881.154618:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790357504:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827881.154632:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827881.154640:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827881/real 1787827881] req@ffff9e6831aa8000 x1874668790357504/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827881.154652:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790357504/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827881.154660:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827881.154663:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827881.154666:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790357504:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827881.665769:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827881.857852:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827882.689769:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827882.881830:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827883.713777:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827883.905844:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827884.737750:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827884.929808:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827885.761769:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827885.953840:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827886.273765:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827886.273775:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827886.273779:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827886.273788:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4522 00000100:00080000:1.0:1787827886.273798:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827886.273805:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827886.273842:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790357632/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827886.273877:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827886.273881:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827886.273883:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827886.273888:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4522 00000100:00080000:1.0:1787827886.273892:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827886.273897:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827886.273908:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827886.273916:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790357632:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787827886.273924:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790357760/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827886.273937:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827886.273941:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827886.273943:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827886.273948:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4522 00000100:00080000:1.0:1787827886.273953:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827886.273957:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827886.273962:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827886.273974:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790357888/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827886.273984:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827886.273989:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827886.273994:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827886.273997:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827886.274002:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4522 00000100:00080000:1.0:1787827886.274007:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827886.274011:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827886.274017:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827886.274031:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827886/real 1787827886] req@ffff9e6831aab480 x1874668790357632/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827886.274033:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790358016/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827886.274057:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827886.274061:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790357632/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827886.274063:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827886.274068:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827886.274075:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827886.274078:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827886.274081:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790357632:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827886.274085:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4522 00000100:00080000:1.0:1787827886.274091:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827886.274096:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827886.274102:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827886.274104:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790357760:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827886.274121:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790358144/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827886.274134:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827886.274136:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827886.274141:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827886.274144:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827886.274150:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4522 00000100:00100000:3.0:1787827886.274154:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827886/real 1787827886] req@ffff9e6831aa9a40 x1874668790357760/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827886.274157:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827886.274167:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827886.274174:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827886.274178:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790357760/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827886.274187:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787827886.274190:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790358272/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827886.274192:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827886.274198:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790357760:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827886.274208:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790357888:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827886.274224:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827886.274228:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827886.274230:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787827886.274233:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827886.274239:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4522 00000100:00100000:3.0:1787827886.274245:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827886/real 1787827886] req@ffff9e6831aaad00 x1874668790357888/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827886.274255:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827886.274259:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790357888/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827886.274266:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:3.0:1787827886.274269:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827886.274274:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:1.0:1787827886.274275:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827886.274277:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790357888:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827886.274284:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790358016:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827886.274294:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790358400/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827886.274300:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827886.274309:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827886/real 1787827886] req@ffff9e6831aabc00 x1874668790358016/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827886.274312:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827886.274324:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787827886.274328:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790358016/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827886.274329:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827886.274340:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4522 00000100:00080000:3.0:1787827886.274340:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827886.274347:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787827886.274349:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827886.274350:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790358016:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787827886.274359:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827886.274366:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827886.274369:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790358144:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827886.274385:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790358528/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827886.274388:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827886.274397:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827886/real 1787827886] req@ffff9e6831aab480 x1874668790358144/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827886.274418:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827886.274424:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787827886.274425:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790358144/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827886.274429:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827886.274437:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827886.274440:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4522 00000100:00080000:3.0:1787827886.274444:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827886.274449:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790358144:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787827886.274450:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827886.274456:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790358272:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827886.274461:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827886.274469:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827886.274484:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790358656/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827886.274490:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827886.274507:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827886/real 1787827886] req@ffff9e6831aa8000 x1874668790358272/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827886.274517:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790358272/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827886.274524:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827886.274526:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827886.274528:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790358272:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827886.274532:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790358400:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827886.274544:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827886.274550:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827886/real 1787827886] req@ffff9e6831aaa1c0 x1874668790358400/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827886.274558:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790358400/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827886.274566:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827886.274569:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827886.274573:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790358400:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827886.274588:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790358528:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827886.274610:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827886.274622:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827886/real 1787827886] req@ffff9e6831aa8780 x1874668790358528/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827886.274635:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790358528/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827886.274644:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827886.274647:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827886.274651:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790358528:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827886.274656:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790358656:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827886.274672:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827886.274680:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827886/real 1787827886] req@ffff9e6831aaa940 x1874668790358656/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827886.274688:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790358656/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827886.274693:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827886.274716:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827886.274719:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790358656:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827886.785766:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827886.977833:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827887.809812:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827888.001854:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827888.833814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827889.025841:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827889.857812:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827890.049883:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827890.881797:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827891.073881:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827891.393746:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827891.393754:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827891.393758:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827891.393766:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4528 00000100:00080000:1.0:1787827891.393775:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827891.393780:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827891.393813:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790358784/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827891.393843:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827891.393845:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827891.393847:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827891.393850:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4528 00000100:00080000:1.0:1787827891.393852:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827891.393854:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827891.393862:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827891.393872:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790358912/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827891.393879:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827891.393881:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827891.393883:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827891.393885:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4528 00000100:00080000:1.0:1787827891.393887:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827891.393889:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827891.393891:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827891.393897:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790359040/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827891.393904:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827891.393907:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827891.393908:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827891.393911:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4528 00000100:00080000:1.0:1787827891.393913:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827891.393916:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827891.393919:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827891.393928:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790359168/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827891.393928:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790358784:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827891.393942:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827891.393944:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827891.393945:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827891.393948:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4528 00000100:00080000:1.0:1787827891.393950:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827891.393952:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827891.393956:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827891.393964:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790359296/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827891.393971:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827891.393974:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827891.393975:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827891.393977:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4528 00000100:00080000:1.0:1787827891.393983:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827891.393985:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827891.393988:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787827891.393993:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827891.394000:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790359424/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827891.394007:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827891.394009:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827891.394011:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827891.394013:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4528 00000100:00080000:1.0:1787827891.394016:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827891.394018:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827891.394021:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827891.394031:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790359552/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827891.394033:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827891/real 1787827891] req@ffff9e6831aa8f00 x1874668790358784/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827891.394052:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827891.394058:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787827891.394059:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790358784/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827891.394064:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827891.394072:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827891.394075:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4528 00000100:00080000:3.0:1787827891.394078:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787827891.394085:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827891.394086:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790358784:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827891.394097:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827891.394098:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790358912:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787827891.394106:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827891.394122:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790359680/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787827891.394123:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827891.394134:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827891.394136:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827891/real 1787827891] req@ffff9e6831aa8b40 x1874668790358912/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827891.394137:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827891.394141:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827891.394150:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4528 00000100:00080000:1.0:1787827891.394156:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827891.394157:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790358912/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827891.394161:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827891.394170:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787827891.394172:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827891.394177:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827891.394181:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790359808/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827891.394183:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790358912:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827891.394197:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790359040:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827891.394221:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827891.394233:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827891/real 1787827891] req@ffff9e6831aa9680 x1874668790359040/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827891.394247:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790359040/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827891.394256:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827891.394259:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827891.394262:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790359040:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827891.394283:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790359168:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827891.394299:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827891.394307:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827891/real 1787827891] req@ffff9e6831aaa580 x1874668790359168/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827891.394314:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790359168/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827891.394320:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827891.394322:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827891.394324:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790359168:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827891.394327:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790359296:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827891.394343:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827891.394349:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827891/real 1787827891] req@ffff9e6831aa9e00 x1874668790359296/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827891.394356:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790359296/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827891.394362:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827891.394377:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827891.394379:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790359296:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827891.394384:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790359424:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827891.394398:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827891.394406:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827891/real 1787827891] req@ffff9e6831aaa940 x1874668790359424/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827891.394413:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790359424/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827891.394419:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827891.394422:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827891.394424:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790359424:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827891.394427:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790359552:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827891.394438:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827891.394443:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827891/real 1787827891] req@ffff9e6831aa8780 x1874668790359552/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827891.394450:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790359552/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827891.394455:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827891.394457:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827891.394459:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790359552:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827891.394463:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790359680:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827891.394472:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827891.394477:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827891/real 1787827891] req@ffff9e6831aaa1c0 x1874668790359680/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827891.394484:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790359680/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827891.394489:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827891.394490:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827891.394492:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790359680:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827891.394495:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790359808:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827891.394505:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827891.394511:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827891/real 1787827891] req@ffff9e6831aa8000 x1874668790359808/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827891.394517:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790359808/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827891.394521:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827891.394523:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827891.394525:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790359808:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827891.905798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827892.097945:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827892.929785:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827893.121860:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827893.953812:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827894.145970:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827894.977777:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827895.169920:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827896.001759:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827896.193844:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827896.513785:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827896.513795:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827896.513799:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827896.513809:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4533 00000100:00080000:2.0:1787827896.513820:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827896.513828:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827896.513865:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790359936/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827896.513892:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827896.513896:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827896.513898:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827896.513902:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4533 00000100:00080000:2.0:1787827896.513907:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827896.513912:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787827896.513922:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827896.513928:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790359936:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787827896.513938:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790360064/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827896.513957:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827896.513960:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827896.513963:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827896.513967:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4533 00000100:00080000:2.0:1787827896.513971:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827896.513976:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787827896.513982:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787827896.513985:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827896.514001:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790360192/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827896.514013:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827896.514017:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827896.514022:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827896.514022:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827896/real 1787827896] req@ffff9e6804420780 x1874668790359936/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827896.514031:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4533 00000100:00080000:2.0:1787827896.514039:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827896.514044:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790359936/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827896.514047:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787827896.514053:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787827896.514066:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787827896.514071:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790360320/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827896.514073:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827896.514080:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790359936:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787827896.514094:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827896.514098:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827896.514100:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827896.514105:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790360064:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787827896.514107:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4533 00000100:00080000:2.0:1787827896.514116:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827896.514122:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787827896.514128:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787827896.514129:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827896.514141:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827896/real 1787827896] req@ffff9e6804422940 x1874668790360064/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827896.514148:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790360448/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827896.514154:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790360064/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827896.514175:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827896.514177:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827896.514179:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827896.514185:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790360064:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787827896.514186:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827896.514192:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787827896.514193:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790360192:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787827896.514202:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4533 00000100:00080000:2.0:1787827896.514209:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787827896.514211:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787827896.514219:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827896.514223:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827896/real 1787827896] req@ffff9e6804421e00 x1874668790360192/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787827896.514227:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827896.514246:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790360576/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827896.514246:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790360192/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827896.514268:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827896.514273:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827896.514278:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790360192:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787827896.514279:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787827896.514284:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790360320:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787827896.514287:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787827896.514290:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827896.514297:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4533 00000800:00000100:3.0:1787827896.514302:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787827896.514305:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787827896.514311:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827896/real 1787827896] req@ffff9e6804422d00 x1874668790360320/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827896.514314:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787827896.514326:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827896.514341:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790360320/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827896.514352:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827896.514355:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827896.514357:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790360320:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827896.514363:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790360704/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827896.514373:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790360448:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787827896.514391:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827896.514397:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1787827896.514398:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787827896.514402:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827896.514408:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4533 00000100:00100000:3.0:1787827896.514412:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827896/real 1787827896] req@ffff9e6804420f00 x1874668790360448/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827896.514415:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827896.514424:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827896.514429:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790360448/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787827896.514432:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827896.514449:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790360832/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827896.514455:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827896.514465:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827896.514465:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787827896.514473:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787827896.514474:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790360448:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787827896.514476:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827896.514485:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4533 00000100:00100000:3.0:1787827896.514488:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790360576:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787827896.514490:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827896.514497:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787827896.514502:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787827896.514513:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827896.514517:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790360960/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787827896.514526:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827896/real 1787827896] req@ffff9e6804420780 x1874668790360576/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827896.514545:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790360576/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827896.514553:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827896.514555:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827896.514557:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790360576:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827896.514567:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790360704:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787827896.514579:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827896.514586:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827896/real 1787827896] req@ffff9e6804421680 x1874668790360704/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827896.514593:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790360704/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827896.514598:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827896.514600:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827896.514603:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790360704:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827896.514606:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790360832:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827896.514616:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827896.514622:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827896/real 1787827896] req@ffff9e68044212c0 x1874668790360832/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827896.514628:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790360832/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827896.514633:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827896.514635:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827896.514638:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790360832:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827896.514643:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790360960:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787827896.514660:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827896.514671:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827896/real 1787827896] req@ffff9e6804423840 x1874668790360960/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827896.514684:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790360960/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827896.514712:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827896.514715:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827896.514718:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790360960:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827897.025770:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827897.217834:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827898.049764:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827898.241821:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827899.073768:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827899.265818:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827900.097796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827900.289928:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827901.121762:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827901.313841:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827901.633792:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827901.633800:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827901.633803:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827901.633811:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4538 00000100:00080000:3.0:1787827901.633820:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827901.633825:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827901.633857:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790361088/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827901.633881:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827901.633884:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827901.633885:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827901.633889:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4538 00000100:00080000:3.0:1787827901.633892:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827901.633895:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827901.633903:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827901.633911:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790361216/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827901.633916:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790361088:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827901.633923:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827901.633926:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827901.633927:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827901.633931:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4538 00000100:00080000:3.0:1787827901.633933:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827901.633935:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827901.633939:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827901.633946:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790361344/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827901.633953:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827901.633955:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827901.633957:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827901.633959:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4538 00000100:00080000:3.0:1787827901.633963:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827901.633966:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827901.633969:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827901.633976:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790361472/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827901.633982:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827901.633985:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827901.633986:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827901.633989:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4538 00000100:00080000:3.0:1787827901.633991:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827901.633994:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827901.634006:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827901.634014:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790361600/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827901.634020:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827901.634022:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827901.634024:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827901.634028:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790361216:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827901.634030:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4538 00000100:00080000:3.0:1787827901.634036:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827901.634040:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827901.634044:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827901.634054:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790361728/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827901.634063:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790361344:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827901.634072:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827901.634077:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827901.634079:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827901.634083:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4538 00000100:00080000:3.0:1787827901.634087:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827901.634090:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827901.634094:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790361472:192.168.203.109@tcp:8:kworker.0 00000008:00100000:3.0:1787827901.634094:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827901.634109:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790361856/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827901.634116:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827901.634119:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827901.634120:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827901.634122:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4538 00000100:00080000:3.0:1787827901.634125:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827901.634126:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790361600:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787827901.634130:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827901.634137:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827901.634147:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790361984/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827901.634156:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827901.634159:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790361728:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827901.634160:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827901.634164:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827901.634169:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4538 00000100:00080000:3.0:1787827901.634173:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827901.634176:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827901.634181:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827901.634184:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790361856:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827901.634191:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790362112/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827901.634217:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790361984:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827901.634230:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790362112:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827902.145758:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827902.337865:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827903.169768:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827903.361853:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827904.193754:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827904.385860:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787827904.705974:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:1.0:1787827904.706010:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787827904.706023:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787827904.706066:0:2386:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:1.0:1787827904.706079:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787827904.706090:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787827904.706100:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787827904.706107:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827901/real 1787827904] req@ffff9e680a2d52c0 x1874668790362112/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787827904.706112:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787827904.706128:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787827904.706139:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790362112/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787827904.706144:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787827904.706155:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00080000:2.0:1787827904.706162:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827904.706170:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827904.706175:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790362112:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827904.706184:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827901/real 1787827904] req@ffff9e680a2d7c00 x1874668790361728/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827904.706200:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790361728/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827904.706210:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827904.706214:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827904.706217:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790361728:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827904.706224:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827901/real 1787827904] req@ffff9e680a2d6d00 x1874668790361856/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827904.706235:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790361856/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827904.706243:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827904.706246:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827904.706249:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790361856:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827904.706255:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827901/real 1787827904] req@ffff9e680a2d70c0 x1874668790361216/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827904.706267:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790361216/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827904.706296:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827904.706300:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827904.706304:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790361216:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827904.706312:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827901/real 1787827904] req@ffff9e680a2d7480 x1874668790361344/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827904.706327:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790361344/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827904.706335:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827904.706339:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827904.706342:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790361344:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827904.706348:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827901/real 1787827904] req@ffff9e680a2d4f00 x1874668790361472/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827904.706359:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790361472/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827904.706366:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827904.706369:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827904.706372:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790361472:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827904.706378:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827901/real 1787827904] req@ffff9e680a2d4b40 x1874668790361600/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827904.706388:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790361600/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827904.706395:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827904.706398:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827904.706401:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790361600:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827904.706407:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827901/real 1787827904] req@ffff9e680a2d43c0 x1874668790361088/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827904.706419:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790361088/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827904.706425:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827904.706428:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827904.706431:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790361088:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787827904.706465:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827901/real 1787827904] req@ffff9e680a2d6580 x1874668790361984/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827904.706475:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790361984/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827904.706483:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827904.706487:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827904.706490:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790361984:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827905.217784:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827905.409889:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827906.241795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827906.433865:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827906.753753:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827906.753762:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827906.753765:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827906.753773:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4543 00000100:00080000:3.0:1787827906.753782:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827906.753788:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827906.753819:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790362240/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827906.753846:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827906.753849:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827906.753851:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827906.753854:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4543 00000100:00080000:3.0:1787827906.753856:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827906.753859:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827906.753868:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827906.753878:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790362368/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827906.753885:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827906.753888:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827906.753889:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827906.753892:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4543 00000100:00080000:3.0:1787827906.753894:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827906.753898:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790362240:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827906.753899:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827906.753906:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827906.753916:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790362496/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827906.753926:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827906.753928:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827906.753929:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827906.753932:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4543 00000100:00080000:3.0:1787827906.753934:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827906.753937:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827906.753941:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827906.753950:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790362624/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827906.753957:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827906.753960:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827906.753961:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827906.753964:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4543 00000100:00080000:3.0:1787827906.753966:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827906.753969:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827906.753972:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827906.753974:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827906.753985:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790362752/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827906.753994:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827906.753996:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827906.753999:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827906.754003:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4543 00000100:00080000:3.0:1787827906.754009:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827906.754014:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827906/real 1787827906] req@ffff9e680a2d4000 x1874668790362240/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827906.754022:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827906.754030:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827906.754034:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790362240/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827906.754045:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787827906.754049:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790362880/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827906.754052:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827906.754061:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790362240:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827906.754067:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827906.754072:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787827906.754075:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790362368:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827906.754077:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827906.754087:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4543 00000100:00080000:3.0:1787827906.754093:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827906.754098:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827906.754105:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827906.754113:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827906.754122:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790363008/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827906.754133:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827906.754138:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827906.754140:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827906.754143:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827906/real 1787827906] req@ffff9e680a2d61c0 x1874668790362368/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827906.754152:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4543 00000100:00080000:3.0:1787827906.754158:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827906.754163:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790362368/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827906.754166:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827906.754174:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787827906.754178:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827906.754185:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827906.754190:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790363136/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827906.754192:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790362368:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827906.754208:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827906.754212:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827906.754214:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827906.754219:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4543 00000100:00080000:3.0:1787827906.754224:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827906.754228:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790362496:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827906.754230:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827906.754239:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827906.754272:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790363264/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827906.754281:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827906.754302:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827906/real 1787827906] req@ffff9e680a2d5e00 x1874668790362496/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827906.754315:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790362496/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827906.754324:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827906.754328:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827906.754331:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790362496:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827906.754337:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790362624:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827906.754356:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827906.754367:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827906/real 1787827906] req@ffff9e680a2d4780 x1874668790362624/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827906.754378:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790362624/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827906.754385:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827906.754389:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827906.754392:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790362624:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827906.754398:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790362752:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827906.754421:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827906.754431:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827906/real 1787827906] req@ffff9e680a2d5a40 x1874668790362752/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827906.754442:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790362752/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827906.754450:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827906.754453:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827906.754456:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790362752:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827906.754462:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790362880:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827906.754482:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827906.754493:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827906/real 1787827906] req@ffff9e680a2d6580 x1874668790362880/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827906.754514:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790362880/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827906.754524:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827906.754527:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827906.754531:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790362880:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827906.754537:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790363008:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827906.754558:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827906.754570:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827906/real 1787827906] req@ffff9e680a2d43c0 x1874668790363008/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827906.754584:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790363008/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827906.754592:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827906.754596:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827906.754600:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790363008:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827906.754606:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790363136:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827906.754627:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827906.754639:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827906/real 1787827906] req@ffff9e680a2d4b40 x1874668790363136/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827906.754652:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790363136/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827906.754669:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827906.754672:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827906.754676:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790363136:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827906.754731:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790363264:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827906.754755:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827906.754768:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827906/real 1787827906] req@ffff9e680a2d4f00 x1874668790363264/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827906.754780:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790363264/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827906.754788:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827906.754792:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827906.754795:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790363264:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827907.265757:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827907.457861:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827908.289776:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827908.481864:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827909.313753:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827909.505884:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827910.337804:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827910.529861:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827911.361769:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827911.553960:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827911.873774:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827911.873785:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827911.873789:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827911.873801:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4548 00000100:00080000:3.0:1787827911.873813:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827911.873820:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827911.873865:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790363392/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827911.873891:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827911.873894:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827911.873896:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827911.873899:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4548 00000100:00080000:3.0:1787827911.873901:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827911.873904:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827911.873914:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827911.873923:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790363520/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827911.873932:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790363392:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827911.873934:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827911.873942:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827911.873944:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827911.873947:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4548 00000100:00080000:3.0:1787827911.873949:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827911.873951:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827911.873954:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827911.873962:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790363648/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827911.873968:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827911.873971:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827911.873972:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827911.873975:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4548 00000100:00080000:3.0:1787827911.873977:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827911.873979:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827911.873982:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827911.873989:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790363776/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827911.873994:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827911.873997:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827911.873998:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827911.874000:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4548 00000100:00080000:3.0:1787827911.874003:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827911.874006:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827911.874010:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827911.874014:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827911.874022:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790363904/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827911.874030:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827911.874033:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827911.874035:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827911.874037:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4548 00000100:00080000:3.0:1787827911.874040:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827911.874043:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827911.874045:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827911.874054:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790364032/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827911.874060:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827911.874064:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827911/real 1787827911] req@ffff9e680a2d7480 x1874668790363392/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827911.874065:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827911.874068:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827911.874078:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4548 00000100:00080000:3.0:1787827911.874083:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827911.874086:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827911.874089:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827911.874092:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790363392/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827911.874100:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790364160/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827911.874124:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827911.874125:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827911.874129:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827911.874134:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827911.874134:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827911.874142:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790363392:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827911.874149:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4548 00000100:00080000:3.0:1787827911.874152:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827911.874156:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827911.874159:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827911.874168:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790364288/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827911.874170:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790363520:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827911.874186:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827911.874189:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827911.874191:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827911.874194:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4548 00000100:00080000:3.0:1787827911.874197:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827911.874200:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827911.874204:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827911.874208:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827911.874234:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827911/real 1787827911] req@ffff9e680a2d70c0 x1874668790363520/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827911.874236:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790364416/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827911.874275:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790363520/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827911.874283:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827911.874286:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827911.874288:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790363520:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827911.874293:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790363648:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827911.874311:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827911.874320:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827911/real 1787827911] req@ffff9e680a2d6d00 x1874668790363648/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827911.874327:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790363648/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827911.874333:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827911.874335:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827911.874337:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790363648:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827911.874342:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790363776:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827911.874354:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827911.874360:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827911/real 1787827911] req@ffff9e680a2d7c00 x1874668790363776/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827911.874367:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790363776/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827911.874373:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827911.874374:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827911.874376:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790363776:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827911.874380:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790363904:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827911.874390:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827911.874396:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827911/real 1787827911] req@ffff9e680a2d52c0 x1874668790363904/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827911.874403:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790363904/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827911.874408:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827911.874410:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827911.874412:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790363904:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827911.874415:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790364032:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827911.874426:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827911.874431:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827911/real 1787827911] req@ffff9e680a2d4f00 x1874668790364032/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827911.874440:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790364032/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827911.874446:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827911.874448:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827911.874450:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790364032:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827911.874453:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790364160:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827911.874464:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827911.874470:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827911/real 1787827911] req@ffff9e680a2d4b40 x1874668790364160/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827911.874477:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790364160/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827911.874482:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827911.874484:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827911.874485:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790364160:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827911.874501:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790364288:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827911.874511:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827911.874517:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827911/real 1787827911] req@ffff9e680a2d43c0 x1874668790364288/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827911.874525:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790364288/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827911.874535:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827911.874537:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827911.874539:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790364288:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827911.874543:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790364416:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827911.874552:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827911.874558:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827911/real 1787827911] req@ffff9e680a2d6580 x1874668790364416/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827911.874565:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790364416/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827911.874570:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827911.874572:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827911.874573:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790364416:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827912.385767:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827912.577889:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827913.409794:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827913.601943:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827914.433782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827914.625868:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827915.457769:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827915.649845:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827916.481764:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827916.673896:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827916.993796:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827916.993804:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827916.993807:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827916.993815:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4553 00000100:00080000:1.0:1787827916.993825:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827916.993830:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827916.993861:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790364544/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827916.993883:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827916.993886:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827916.993887:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827916.993890:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4553 00000100:00080000:1.0:1787827916.993892:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827916.993895:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827916.993902:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827916.993911:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790364672/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827916.993923:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827916.993926:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827916.993927:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827916.993932:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4553 00000100:00080000:1.0:1787827916.993934:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827916.993934:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790364544:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827916.993940:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827916.993946:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827916.993951:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790364800/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827916.993957:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827916.993960:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827916.993961:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827916.993963:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4553 00000100:00080000:1.0:1787827916.993965:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827916.993968:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827916.993971:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827916.993979:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790364928/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827916.993984:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827916.993986:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827916.993987:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827916.993990:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4553 00000100:00080000:1.0:1787827916.993995:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827916.993997:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827916.994001:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827916.994005:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827916.994012:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790365056/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827916.994020:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827916.994023:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827916.994024:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827916.994027:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4553 00000100:00080000:1.0:1787827916.994029:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827916.994031:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827916.994034:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827916.994043:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790365184/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827916.994046:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827916/real 1787827916] req@ffff9e6831aab480 x1874668790364544/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827916.994058:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827916.994063:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827916.994065:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827916.994070:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4553 00000100:00100000:2.0:1787827916.994071:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790364544/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827916.994073:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827916.994080:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827916.994087:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787827916.994089:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827916.994095:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827916.994096:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790365312/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827916.994101:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790364544:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827916.994116:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827916.994118:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827916.994119:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827916.994122:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4553 00000100:00080000:1.0:1787827916.994124:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827916.994128:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790364672:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827916.994131:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827916.994136:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827916.994145:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790365440/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827916.994153:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787827916.994153:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827916.994158:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827916.994159:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827916.994163:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4553 00000100:00080000:1.0:1787827916.994166:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827916.994167:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827916/real 1787827916] req@ffff9e6831aabc00 x1874668790364672/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827916.994171:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827916.994178:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827916.994184:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790364672/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827916.994203:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790365568/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827916.994210:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827916.994216:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827916.994220:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790364672:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827916.994226:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790364800:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827916.994242:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827916.994250:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827916/real 1787827916] req@ffff9e6831aaad00 x1874668790364800/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827916.994257:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790364800/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827916.994263:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827916.994265:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827916.994267:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790364800:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827916.994270:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790364928:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827916.994280:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827916.994286:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827916/real 1787827916] req@ffff9e6831aa9a40 x1874668790364928/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827916.994293:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790364928/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827916.994298:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827916.994300:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827916.994302:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790364928:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827916.994305:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790365056:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827916.994315:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827916.994320:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827916/real 1787827916] req@ffff9e6831aa8000 x1874668790365056/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827916.994327:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790365056/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827916.994332:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827916.994333:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827916.994335:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790365056:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827916.994338:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790365184:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827916.994348:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827916.994353:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827916/real 1787827916] req@ffff9e6831aaa1c0 x1874668790365184/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827916.994360:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790365184/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827916.994371:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827916.994373:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827916.994375:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790365184:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827916.994378:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790365312:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827916.994387:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827916.994392:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827916/real 1787827916] req@ffff9e6831aa8780 x1874668790365312/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827916.994398:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790365312/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827916.994404:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827916.994405:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827916.994407:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790365312:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827916.994419:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790365440:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827916.994429:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827916.994435:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827916/real 1787827916] req@ffff9e6831aaa940 x1874668790365440/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827916.994442:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790365440/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827916.994447:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827916.994449:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827916.994451:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790365440:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827916.994455:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790365568:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827916.994464:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827916.994470:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827916/real 1787827916] req@ffff9e6831aa9e00 x1874668790365568/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827916.994476:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790365568/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827916.994482:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827916.994483:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827916.994485:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790365568:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827917.505768:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827917.697878:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827918.529763:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827918.721895:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827919.553765:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827919.745862:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827920.577755:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787827920.769820:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827921.601764:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827921.793894:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827922.113768:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827922.113783:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827922.113788:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827922.113798:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4558 00000100:00080000:3.0:1787827922.113810:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827922.113833:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827922.113877:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790365696/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827922.113923:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827922.113927:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827922.113930:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827922.113935:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4558 00000100:00080000:3.0:1787827922.113940:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827922.113949:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827922.113962:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827922.113974:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790365696:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787827922.113979:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790365824/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827922.113998:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827922.114002:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827922.114005:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827922.114011:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4558 00000100:00080000:3.0:1787827922.114014:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827922.114019:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827922.114024:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827922.114033:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790365952/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827922.114040:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827922.114044:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827922.114045:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787827922.114050:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827922.114051:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4558 00000100:00080000:3.0:1787827922.114062:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827922.114066:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827922.114073:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827922.114081:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790366080/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827922.114089:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827922.114093:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827922.114096:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827922.114099:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4558 00000100:00100000:2.0:1787827922.114105:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827922/real 1787827922] req@ffff9e680a2d5a40 x1874668790365696/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827922.114107:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827922.114117:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827922.114130:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827922.114141:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790365696/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827922.114145:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790366208/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827922.114206:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827922.114206:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827922.114218:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827922.114223:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827922.114223:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827922.114245:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790365696:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827922.114245:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4558 00000100:00080000:3.0:1787827922.114263:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827922.114271:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827922.114276:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827922.114287:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790365824:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827922.114292:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790366336/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827922.114311:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827922.114314:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827922.114316:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827922.114320:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4558 00000100:00080000:3.0:1787827922.114324:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827922.114331:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827922.114335:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827922.114343:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827922.114350:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827922/real 1787827922] req@ffff9e680a2d4780 x1874668790365824/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827922.114355:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790366464/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827922.114394:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827922.114395:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790365824/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827922.114402:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827922.114409:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827922.114428:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4558 00000100:00080000:2.0:1787827922.114431:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827922.114433:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827922.114439:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827922.114445:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827922.114452:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790365824:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787827922.114455:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827922.114460:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790365952:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827922.114468:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790366592/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827922.114476:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787827922.114484:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827922.114487:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827922.114491:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827922.114498:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4558 00000100:00100000:2.0:1787827922.114503:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827922/real 1787827922] req@ffff9e680a2d5e00 x1874668790365952/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827922.114505:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827922.114518:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827922.114530:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827922.114538:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790365952/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827922.114547:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790366720/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827922.114551:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827922.114565:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827922.114570:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790365952:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827922.114577:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790366080:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827922.114592:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827922.114598:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827922/real 1787827922] req@ffff9e680a2d61c0 x1874668790366080/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827922.114606:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790366080/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827922.114612:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827922.114614:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827922.114616:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790366080:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827922.114620:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790366208:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827922.114630:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827922.114636:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827922/real 1787827922] req@ffff9e680a2d4000 x1874668790366208/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827922.114643:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790366208/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827922.114648:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827922.114650:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827922.114651:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790366208:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827922.114665:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790366336:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827922.114677:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827922.114683:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827922/real 1787827922] req@ffff9e680a2d5a40 x1874668790366336/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827922.114690:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790366336/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827922.114718:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827922.114720:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827922.114723:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790366336:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827922.114727:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790366464:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827922.114740:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827922.114748:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827922/real 1787827922] req@ffff9e680a2d6580 x1874668790366464/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827922.114755:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790366464/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827922.114760:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827922.114762:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827922.114764:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790366464:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827922.114768:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790366592:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827922.114777:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827922.114782:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827922/real 1787827922] req@ffff9e680a2d43c0 x1874668790366592/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827922.114789:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790366592/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827922.114795:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827922.114797:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827922.114798:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790366592:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827922.114802:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790366720:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827922.114811:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827922.114818:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827922/real 1787827922] req@ffff9e680a2d4b40 x1874668790366720/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827922.114824:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790366720/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827922.114829:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827922.114831:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827922.114833:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790366720:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827922.625797:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827922.818039:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827923.649791:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827923.841897:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827924.673862:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827924.866168:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827925.697782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827925.889830:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827926.721834:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827926.914044:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827927.233828:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827927.233845:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827927.233852:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827927.233868:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4563 00000100:00080000:3.0:1787827927.233886:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827927.233898:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827927.233974:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790366848/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827927.234046:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827927.234051:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827927.234053:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827927.234077:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4563 00000100:00080000:3.0:1787827927.234084:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827927.234091:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827927.234108:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827927.234154:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790366848:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787827927.234156:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790366976/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827927.234190:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827927.234198:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827927.234202:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827927.234210:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4563 00000100:00080000:3.0:1787827927.234218:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827927.234225:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827927.234233:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827927.234249:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790367104/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827927.234267:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827927.234267:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827927.234282:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827927.234288:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827927.234303:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4563 00000100:00080000:3.0:1787827927.234314:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827927.234324:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827927.234337:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827927/real 1787827927] req@ffff9e680a2d4f00 x1874668790366848/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787827927.234338:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827927.234389:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790367232/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827927.234390:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790366848/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827927.234456:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827927.234470:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827927.234482:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790366848:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827927.234484:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827927.234512:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827927.234517:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827927.234526:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4563 00000100:00080000:3.0:1787827927.234533:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827927.234541:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827927.234549:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790366976:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787827927.234569:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827927.234596:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790367360/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827927.234608:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827927.234641:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827927.234650:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827927.234654:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827927.234665:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4563 00000100:00080000:3.0:1787827927.234673:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827927.234678:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827927/real 1787827927] req@ffff9e680a2d52c0 x1874668790366976/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827927.234686:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827927.234760:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827927.234800:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790367488/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827927.234803:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790366976/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827927.234874:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827927.234900:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827927.234909:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827927.234918:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827927.234927:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4563 00000100:00080000:2.0:1787827927.234931:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827927.234941:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790366976:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827927.234944:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827927.234954:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790367104:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827927.234962:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827927.234979:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827927.234993:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827927.235019:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827927/real 1787827927] req@ffff9e680a2d7c00 x1874668790367104/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827927.235036:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790367616/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827927.235044:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790367104/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827927.235093:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827927.235136:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827927.235170:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827927.235179:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790367104:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827927.235182:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827927.235192:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827927.235214:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790367232:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787827927.235225:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4563 00000100:00080000:3.0:1787827927.235235:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827927.235245:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827927.235254:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827927.235264:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827927.235291:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827927/real 1787827927] req@ffff9e680a2d6d00 x1874668790367232/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827927.235296:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790367744/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827927.235381:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827927.235389:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790367232/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827927.235391:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827927.235399:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827927.235423:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4563 00000100:00080000:2.0:1787827927.235424:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827927.235443:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827927.235449:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827927.235451:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790367232:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787827927.235473:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827927.235490:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827927.235495:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790367360:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827927.235534:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790367872/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827927.235557:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827927.235577:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827927/real 1787827927] req@ffff9e680a2d70c0 x1874668790367360/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827927.235595:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790367360/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827927.235606:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827927.235610:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827927.235614:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790367360:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827927.235622:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790367488:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827927.235639:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827927.235649:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827927/real 1787827927] req@ffff9e680a2d7480 x1874668790367488/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827927.235661:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790367488/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827927.235672:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827927.235675:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827927.235678:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790367488:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827927.235684:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790367616:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827927.235770:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827927.235797:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827927/real 1787827927] req@ffff9e680a2d4f00 x1874668790367616/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827927.235822:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790367616/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827927.235839:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827927.235845:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827927.235852:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790367616:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827927.235861:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790367744:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827927.235892:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827927.235906:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827927/real 1787827927] req@ffff9e680a2d4b40 x1874668790367744/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827927.235920:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790367744/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827927.235929:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827927.235933:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827927.235936:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790367744:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827927.235958:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790367872:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827927.235977:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827927.235988:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827927/real 1787827927] req@ffff9e680a2d43c0 x1874668790367872/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827927.236011:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790367872/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827927.236020:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827927.236024:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827927.236027:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790367872:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827927.745828:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827927.937881:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827928.769806:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827928.961849:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827929.793825:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827929.985842:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827930.817874:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827931.009958:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827931.841775:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827932.033908:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827932.353763:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827932.353771:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827932.353775:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827932.353782:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4569 00000100:00080000:3.0:1787827932.353791:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827932.353798:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827932.353841:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790368000/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827932.353862:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827932.353865:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827932.353866:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827932.353869:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4569 00000100:00080000:3.0:1787827932.353878:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827932.353881:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827932.353888:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827932.353898:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790368128/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827932.353906:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827932.353908:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827932.353909:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827932.353911:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4569 00000100:00080000:3.0:1787827932.353913:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827932.353918:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790368000:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827932.353919:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827932.353926:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827932.353933:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790368256/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827932.353941:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827932.353943:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827932.353944:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827932.353948:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4569 00000100:00080000:3.0:1787827932.353950:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827932.353953:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827932.353956:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827932.353963:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790368384/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827932.353970:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827932.353972:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827932.353973:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827932.353975:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4569 00000100:00080000:3.0:1787827932.353978:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827932.353981:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827932.353985:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827932.353990:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827932.353999:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790368512/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827932.354004:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827932.354006:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827932.354007:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827932.354010:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4569 00000100:00080000:3.0:1787827932.354012:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827932.354015:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827932.354018:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827932.354031:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827932/real 1787827932] req@ffff9e680a2d6580 x1874668790368000/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827932.354031:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790368640/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827932.354056:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827932.354062:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827932.354063:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827932.354066:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4569 00000100:00080000:3.0:1787827932.354069:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827932.354074:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827932.354075:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790368000/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787827932.354077:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787827932.354105:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787827932.354108:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790368768/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827932.354113:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827932.354121:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790368000:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827932.354124:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827932.354128:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827932.354130:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827932.354131:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790368128:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827932.354135:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4569 00000100:00080000:3.0:1787827932.354140:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827932.354143:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827932.354146:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827932.354156:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790368896/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827932.354162:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827932.354169:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827932.354172:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827932.354174:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827932.354178:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827932/real 1787827932] req@ffff9e680a2d5a40 x1874668790368128/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827932.354179:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4569 00000100:00080000:3.0:1787827932.354185:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827932.354191:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827932.354197:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827932.354200:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790368128/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827932.354208:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790369024/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827932.354210:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827932.354218:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827932.354225:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790368128:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827932.354247:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790368256:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827932.354260:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827932.354266:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827932/real 1787827932] req@ffff9e680a2d4000 x1874668790368256/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827932.354273:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790368256/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827932.354279:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827932.354280:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827932.354282:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790368256:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827932.354286:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790368384:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827932.354296:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827932.354301:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827932/real 1787827932] req@ffff9e680a2d61c0 x1874668790368384/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827932.354309:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790368384/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827932.354314:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827932.354315:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827932.354317:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790368384:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827932.354321:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790368512:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827932.354330:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827932.354335:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827932/real 1787827932] req@ffff9e680a2d5e00 x1874668790368512/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827932.354342:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790368512/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827932.354347:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827932.354349:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827932.354351:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790368512:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827932.354354:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790368640:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827932.354363:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827932.354368:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827932/real 1787827932] req@ffff9e680a2d4780 x1874668790368640/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827932.354375:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790368640/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827932.354381:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827932.354382:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827932.354384:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790368640:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827932.354388:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790368768:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827932.354397:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827932.354402:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827932/real 1787827932] req@ffff9e680a2d43c0 x1874668790368768/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827932.354408:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790368768/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827932.354418:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827932.354420:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827932.354422:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790368768:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827932.354425:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790368896:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827932.354434:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827932.354439:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827932/real 1787827932] req@ffff9e680a2d4b40 x1874668790368896/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827932.354446:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790368896/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827932.354451:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827932.354454:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827932.354455:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790368896:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827932.354459:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790369024:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827932.354468:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827932.354473:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827932/real 1787827932] req@ffff9e680a2d4f00 x1874668790369024/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827932.354480:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790369024/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827932.354484:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827932.354486:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827932.354489:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790369024:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827932.865747:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827933.057894:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827933.889818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827934.081965:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827934.913819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827935.105915:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827935.937851:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827936.129958:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827936.961852:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827937.153876:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827937.473780:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827937.473789:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827937.473792:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827937.473801:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4574 00000100:00080000:1.0:1787827937.473812:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827937.473819:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827937.473851:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790369152/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827937.473875:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827937.473878:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827937.473880:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827937.473884:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4574 00000100:00080000:1.0:1787827937.473887:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827937.473890:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827937.473899:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827937.473899:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790369152:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787827937.473917:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790369280/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827937.473930:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827937.473934:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827937.473937:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827937.473941:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4574 00000100:00080000:1.0:1787827937.473946:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827937.473951:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827937.473957:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827937.473973:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790369408/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827937.473973:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827937.474000:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827937.474005:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827937.474008:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827937.474013:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4574 00000100:00080000:1.0:1787827937.474019:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827937.474023:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827937.474031:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827937.474041:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827937/real 1787827937] req@ffff9e6831aaa580 x1874668790369152/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827937.474068:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790369536/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827937.474102:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827937.474106:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827937.474110:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827937.474112:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790369152/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827937.474120:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4574 00000100:00080000:1.0:1787827937.474130:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827937.474139:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787827937.474140:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787827937.474146:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787827937.474150:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827937.474157:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790369152:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787827937.474167:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790369664/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827937.474195:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827937.474199:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827937.474201:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827937.474206:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790369280:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827937.474207:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4574 00000100:00080000:1.0:1787827937.474216:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827937.474221:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827937.474228:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827937.474243:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790369792/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827937.474254:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827937.474261:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827937.474266:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827937.474267:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827937.474271:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4574 00000100:00080000:1.0:1787827937.474277:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827937.474281:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827937.474288:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827937.474293:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827937/real 1787827937] req@ffff9e6831aa9680 x1874668790369280/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827937.474303:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790369920/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827937.474317:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790369280/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827937.474320:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827937.474330:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827937.474335:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827937.474346:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4574 00000100:00080000:2.0:1787827937.474347:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827937.474351:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827937.474355:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827937.474359:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790369280:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827937.474373:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827937.474376:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790369408:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787827937.474381:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827937.474401:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790370048/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827937.474411:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827937.474416:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827937.474421:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827937.474424:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827937.474430:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827937/real 1787827937] req@ffff9e6831aa8b40 x1874668790369408/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827937.474431:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4574 00000100:00080000:1.0:1787827937.474447:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827937.474453:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790369408/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827937.474458:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827937.474469:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787827937.474472:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827937.474479:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827937.474487:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790370176/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827937.474489:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790369408:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827937.474529:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790369536:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827937.474555:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827937.474568:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827937/real 1787827937] req@ffff9e6831aa8f00 x1874668790369536/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827937.474577:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790369536/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827937.474583:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827937.474585:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827937.474589:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790369536:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827937.474595:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790369664:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827937.474616:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827937.474628:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827937/real 1787827937] req@ffff9e6831aa9e00 x1874668790369664/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827937.474642:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790369664/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827937.474648:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827937.474651:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827937.474653:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790369664:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827937.474667:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790369792:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827937.474682:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827937.474690:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827937/real 1787827937] req@ffff9e6831aaa940 x1874668790369792/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827937.474701:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790369792/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827937.474735:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827937.474740:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827937.474744:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790369792:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827937.474751:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790369920:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827937.474768:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827937.474776:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827937/real 1787827937] req@ffff9e6831aa8780 x1874668790369920/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827937.474800:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790369920/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827937.474808:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827937.474811:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827937.474815:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790369920:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827937.474821:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790370048:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827937.474838:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827937.474846:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827937/real 1787827937] req@ffff9e6831aaa1c0 x1874668790370048/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827937.474856:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790370048/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827937.474864:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827937.474866:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827937.474869:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790370048:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827937.474874:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790370176:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827937.474888:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827937.474896:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827937/real 1787827937] req@ffff9e6831aa8000 x1874668790370176/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827937.474911:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790370176/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827937.474923:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827937.474927:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827937.474947:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790370176:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827937.985786:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827938.177921:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827939.009836:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827939.201878:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827940.033783:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827940.225853:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827941.057809:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827941.249849:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827942.081765:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827942.273957:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827942.593862:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827942.593873:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827942.593878:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827942.593889:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4579 00000100:00080000:3.0:1787827942.593913:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827942.593922:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827942.593963:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790370304/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827942.594026:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827942.594049:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827942.594054:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827942.594072:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4579 00000100:00080000:3.0:1787827942.594082:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827942.594090:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827942.594099:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790370304:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787827942.594110:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827942.594142:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790370432/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827942.594163:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827942.594168:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827942.594172:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827942.594178:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4579 00000100:00080000:3.0:1787827942.594186:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827942.594191:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827942.594200:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827942.594206:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827942.594227:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790370560/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827942.594289:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827942/real 1787827942] req@ffff9e680a2d7480 x1874668790370304/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827942.594340:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790370304/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827942.594365:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827942.594370:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827942.594379:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790370304:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827942.594387:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827942.594396:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827942.594401:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827942.594410:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4579 00000100:00100000:1.0:1787827942.594419:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790370432:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827942.594419:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827942.594438:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827942.594451:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787827942.594487:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827942.594519:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827942/real 1787827942] req@ffff9e680a2d70c0 x1874668790370432/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827942.594536:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790370432/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827942.594577:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827942.594581:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827942.594585:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790370432:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827942.594593:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790370560:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827942.594623:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827942.594638:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827942/real 1787827942] req@ffff9e680a2d6d00 x1874668790370560/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827942.594652:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790370560/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827942.594662:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827942.594665:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827942.594669:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790370560:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827942.594785:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790370688/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827942.594847:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827942.594854:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827942.594859:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827942.594870:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4579 00000100:00080000:3.0:1787827942.594881:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827942.594906:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827942.594918:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790370688:192.168.203.109@tcp:8:kworker.0 00000008:00100000:3.0:1787827942.594924:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787827942.594951:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827942.594976:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827942/real 1787827942] req@ffff9e680a2d7c00 x1874668790370688/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827942.595006:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790370688/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827942.595036:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790370816/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827942.595038:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827942.595069:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827942.595077:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827942.595084:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790370688:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787827942.595086:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827942.595094:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827942.595102:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4579 00000100:00080000:3.0:1787827942.595109:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827942.595117:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790370816:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787827942.595118:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827942.595137:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827942.595163:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790370944/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827942.595170:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827942.595192:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827942.595193:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827942/real 1787827942] req@ffff9e680a2d52c0 x1874668790370816/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827942.595210:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827942.595218:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827942.595226:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790370816/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827942.595231:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4579 00000100:00080000:3.0:1787827942.595245:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827942.595259:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827942.595259:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787827942.595276:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:3.0:1787827942.595276:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827942.595290:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790370816:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827942.595305:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790371072/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827942.595321:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790370944:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827942.595350:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827942.595356:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827942.595360:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827942.595368:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4579 00000100:00080000:3.0:1787827942.595375:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827942.595379:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827942.595389:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827942.595401:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827942.595412:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827942/real 1787827942] req@ffff9e680a2d7c00 x1874668790370944/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827942.595421:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790371200/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827942.595462:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827942.595468:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787827942.595487:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790370944/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827942.595488:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827942.595507:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827942.595507:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4579 00000100:00080000:3.0:1787827942.595516:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827942.595517:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827942.595523:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827942.595525:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790370944:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787827942.595529:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827942.595541:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790371328/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827942.595544:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790371072:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827942.595604:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827942.595621:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827942/real 1787827942] req@ffff9e680a2d6d00 x1874668790371072/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827942.595643:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790371072/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827942.595667:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827942.595673:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827942.595679:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790371072:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827942.595690:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790371200:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827942.595729:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827942.595806:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827942/real 1787827942] req@ffff9e680a2d70c0 x1874668790371200/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827942.595824:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790371200/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827942.595835:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827942.595850:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827942.595855:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790371200:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827942.595869:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790371328:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827942.595888:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827942.595898:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827942/real 1787827942] req@ffff9e680a2d7480 x1874668790371328/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827942.595907:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790371328/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827942.595914:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827942.595916:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827942.595918:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790371328:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827943.105797:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827943.298111:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827944.129751:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827944.321968:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827945.153771:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827945.345900:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827946.177768:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827946.369871:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827947.201760:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827947.393995:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827947.713795:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827947.713810:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827947.713817:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827947.713829:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4584 00000100:00080000:3.0:1787827947.713841:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827947.713848:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827947.713894:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790371456/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827947.713926:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827947.713932:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827947.713934:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827947.713939:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4584 00000100:00080000:3.0:1787827947.713942:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827947.713946:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827947.713957:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827947.713986:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790371456:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787827947.713988:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790371584/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827947.714020:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827947.714025:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827947.714028:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827947.714034:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4584 00000100:00080000:3.0:1787827947.714040:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827947.714046:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827947.714053:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827947.714066:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827947.714073:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790371712/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827947.714089:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827947.714094:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827947.714097:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827947.714104:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4584 00000100:00100000:1.0:1787827947.714106:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827947/real 1787827947] req@ffff9e680a2d4f00 x1874668790371456/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827947.714115:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827947.714127:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790371456/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827947.714127:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827947.714144:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787827947.714147:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827947.714154:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827947.714158:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790371456:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787827947.714163:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790371840/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827947.714176:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827947.714178:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827947.714181:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827947.714185:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4584 00000100:00080000:3.0:1787827947.714191:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827947.714204:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790371584:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827947.714204:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827947.714223:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787827947.714253:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827947.714255:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790371968/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827947.714267:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827947/real 1787827947] req@ffff9e680a2d4b40 x1874668790371584/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827947.714282:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827947.714289:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787827947.714291:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790371584/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827947.714292:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827947.714301:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4584 00000100:00080000:3.0:1787827947.714309:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827947.714317:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787827947.714321:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787827947.714323:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787827947.714332:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827947.714341:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790372096/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827947.714342:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790371584:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827947.714369:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790371712:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827947.714371:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827947.714385:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827947.714388:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827947.714393:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4584 00000100:00080000:3.0:1787827947.714399:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827947.714403:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827947.714408:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827947.714416:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827947.714424:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790372224/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827947.714434:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827947.714441:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787827947.714442:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827947/real 1787827947] req@ffff9e680a2d43c0 x1874668790371712/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827947.714443:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827947.714451:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4584 00000100:00080000:3.0:1787827947.714462:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827947.714473:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827947.714477:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827947.714479:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790371712/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827947.714492:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787827947.714498:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790372352/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827947.714499:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827947.714509:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790371712:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827947.714529:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790371840:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787827947.714542:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827947.714546:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827947.714549:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827947.714554:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4584 00000800:00000100:1.0:1787827947.714562:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827947.714563:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827947.714571:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827947.714577:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827947/real 1787827947] req@ffff9e680a2d4780 x1874668790371840/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787827947.714578:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827947.714604:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790372480/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827947.714613:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790371840/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827947.714627:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827947.714630:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827947.714633:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790371840:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827947.714655:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790371968:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827947.714671:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827947.714683:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827947/real 1787827947] req@ffff9e680a2d5e00 x1874668790371968/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827947.714722:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790371968/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827947.714729:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827947.714731:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827947.714733:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790371968:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827947.714738:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790372096:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827947.714754:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827947.714762:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827947/real 1787827947] req@ffff9e680a2d61c0 x1874668790372096/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827947.714769:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790372096/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827947.714776:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827947.714778:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827947.714780:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790372096:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827947.714784:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790372224:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787827947.714794:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827947.714800:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827947/real 1787827947] req@ffff9e680a2d4000 x1874668790372224/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827947.714807:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790372224/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827947.714817:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827947.714819:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827947.714821:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790372224:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827947.714825:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790372352:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827947.714835:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827947.714840:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827947/real 1787827947] req@ffff9e680a2d5a40 x1874668790372352/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827947.714847:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790372352/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827947.714852:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827947.714854:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827947.714856:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790372352:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827947.714860:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790372480:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787827947.714869:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827947.714875:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827947/real 1787827947] req@ffff9e680a2d6580 x1874668790372480/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827947.714881:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790372480/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827947.714886:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827947.714888:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787827947.714898:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790372480:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827948.225751:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827948.417854:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827949.249781:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827949.441854:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827950.273797:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827950.465884:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827951.297805:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827951.489833:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827952.321779:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827952.513835:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827952.833779:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827952.833787:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827952.833791:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827952.833799:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4589 00000100:00080000:1.0:1787827952.833808:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827952.833814:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827952.833845:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790372608/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827952.833867:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827952.833869:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827952.833871:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827952.833873:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4589 00000100:00080000:1.0:1787827952.833876:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827952.833878:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827952.833886:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827952.833896:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790372736/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827952.833904:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827952.833906:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790372608:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827952.833908:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827952.833912:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827952.833917:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4589 00000100:00080000:1.0:1787827952.833919:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827952.833922:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827952.833925:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827952.833933:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790372864/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827952.833953:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827952.833956:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827952.833958:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827952.833961:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4589 00000100:00080000:1.0:1787827952.833964:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827952.833967:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827952.833971:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827952.833981:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827952.833982:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790372992/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827952.833991:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827952.833993:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827952.833995:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827952.833998:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4589 00000100:00080000:1.0:1787827952.834001:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827952.834004:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827952.834009:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827952.834022:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827952/real 1787827952] req@ffff9e6831aa9a40 x1874668790372608/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827952.834025:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790373120/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827952.834047:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827952.834049:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790372608/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827952.834057:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827952.834063:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827952.834064:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827952.834072:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787827952.834073:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4589 00000100:00100000:2.0:1787827952.834077:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790372608:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827952.834082:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827952.834090:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827952.834096:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827952.834104:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790372736:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827952.834112:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790373248/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827952.834125:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827952.834126:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827952.834131:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827952.834134:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827952.834138:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827952/real 1787827952] req@ffff9e6831aaad00 x1874668790372736/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827952.834143:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4589 00000100:00080000:1.0:1787827952.834152:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827952.834153:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790372736/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827952.834163:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787827952.834166:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827952.834172:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:1.0:1787827952.834174:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827952.834176:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790372736:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827952.834185:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790372864:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827952.834196:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790373376/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827952.834201:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827952.834210:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827952/real 1787827952] req@ffff9e6831aabc00 x1874668790372864/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827952.834216:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827952.834221:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787827952.834223:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790372864/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827952.834224:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827952.834233:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4589 00000100:00080000:1.0:1787827952.834239:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827952.834246:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787827952.834246:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827952.834252:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:1.0:1787827952.834254:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827952.834256:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790372864:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827952.834263:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790372992:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787827952.834269:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790373504/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827952.834277:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787827952.834281:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827952.834283:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827952.834286:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827952.834291:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4589 00000100:00100000:2.0:1787827952.834292:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827952/real 1787827952] req@ffff9e6831aab480 x1874668790372992/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827952.834294:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827952.834301:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827952.834309:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827952.834311:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790372992/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827952.834318:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787827952.834321:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790373632/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827952.834324:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827952.834331:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790372992:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827952.834342:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790373120:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827952.834356:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827952.834363:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827952/real 1787827952] req@ffff9e6831aa8000 x1874668790373120/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827952.834370:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790373120/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827952.834375:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827952.834377:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827952.834379:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790373120:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827952.834389:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790373248:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827952.834399:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827952.834405:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827952/real 1787827952] req@ffff9e6831aaa1c0 x1874668790373248/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827952.834411:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790373248/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827952.834417:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827952.834420:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827952.834423:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790373248:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827952.834428:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790373376:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827952.834447:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827952.834459:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827952/real 1787827952] req@ffff9e6831aa8780 x1874668790373376/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827952.834472:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790373376/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827952.834487:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827952.834490:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827952.834493:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790373376:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827952.834497:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790373504:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827952.834512:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827952.834520:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827952/real 1787827952] req@ffff9e6831aaa940 x1874668790373504/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827952.834528:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790373504/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827952.834533:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827952.834535:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827952.834537:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790373504:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827952.834540:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790373632:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827952.834550:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827952.834555:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827952/real 1787827952] req@ffff9e6831aa9e00 x1874668790373632/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827952.834562:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790373632/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827952.834567:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827952.834569:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827952.834570:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790373632:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827953.345772:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827953.537982:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827954.369815:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827954.561946:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827955.393797:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827955.585946:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827956.417831:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827956.609991:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827957.441790:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827957.634043:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827957.953967:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827957.953980:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827957.953986:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827957.954014:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4594 00000100:00080000:0.0:1787827957.954029:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827957.954038:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827957.954084:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790373760/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827957.954121:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827957.954126:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827957.954128:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827957.954133:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4594 00000100:00080000:0.0:1787827957.954137:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827957.954141:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827957.954153:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827957.954169:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790373888/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827957.954183:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827957.954191:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787827957.954211:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790373760:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787827957.954221:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827957.954228:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4594 00000100:00080000:0.0:1787827957.954231:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827957.954235:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827957.954241:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827957.954256:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790374016/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827957.954265:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827957.954269:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827957.954271:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827957.954274:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4594 00000100:00080000:0.0:1787827957.954277:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827957.954281:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787827957.954288:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787827957.954291:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827957.954305:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790374144/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827957.954314:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827957.954318:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827957.954319:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827957.954323:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4594 00000100:00080000:0.0:1787827957.954327:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827957.954334:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827957.954335:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827957/real 1787827957] req@ffff9e6803e9b840 x1874668790373760/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787827957.954339:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827957.954354:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790374272/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827957.954365:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827957.954365:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790373760/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827957.954370:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827957.954375:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827957.954388:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4594 00000100:00080000:0.0:1787827957.954393:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827957.954393:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827957.954406:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787827957.954407:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827957.954416:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827957.954418:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790373760:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827957.954433:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790374400/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827957.954446:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827957.954451:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790373888:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827957.954453:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827957.954459:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827957.954465:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4594 00000100:00080000:0.0:1787827957.954469:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827957.954473:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827957.954478:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827957.954487:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827957.954493:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790374528/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827957.954503:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827957.954504:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827957/real 1787827957] req@ffff9e6803e98f00 x1874668790373888/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827957.954509:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827957.954514:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827957.954526:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4594 00000100:00080000:0.0:1787827957.954531:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827957.954532:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790373888/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827957.954540:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827957.954552:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787827957.954554:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827957.954564:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787827957.954572:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790374656/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827957.954573:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790373888:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827957.954589:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790374016:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827957.954595:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827957.954600:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827957.954602:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827957.954608:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4594 00000100:00080000:0.0:1787827957.954613:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827957.954618:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827957.954620:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827957.954629:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827957.954635:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827957/real 1787827957] req@ffff9e6803e99e00 x1874668790374016/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827957.954644:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790374784/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827957.954653:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790374016/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827957.954682:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827957.954686:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827957.954691:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790374016:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827957.954700:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790374144:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827957.954766:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827957.954783:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827957/real 1787827957] req@ffff9e6803e99680 x1874668790374144/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827957.954800:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790374144/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827957.954812:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827957.954815:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827957.954818:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790374144:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827957.954823:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790374272:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827957.954843:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827957.954853:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827957/real 1787827957] req@ffff9e6803e9a940 x1874668790374272/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827957.954863:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790374272/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827957.954870:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827957.954873:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827957.954875:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790374272:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827957.954891:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790374400:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827957.954949:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827957.954965:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827957/real 1787827957] req@ffff9e6803e9a580 x1874668790374400/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827957.954980:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790374400/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827957.954991:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827957.954994:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827957.954998:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790374400:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827957.955004:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790374528:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827957.955026:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827957.955036:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827957/real 1787827957] req@ffff9e6803e9a1c0 x1874668790374528/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827957.955049:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790374528/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827957.955058:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827957.955061:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827957.955064:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790374528:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827957.955070:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790374656:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827957.955086:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827957.955096:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827957/real 1787827957] req@ffff9e6803e9b480 x1874668790374656/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827957.955109:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790374656/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827957.955126:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827957.955129:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827957.955133:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790374656:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827957.955139:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790374784:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827957.955155:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827957.955165:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827957/real 1787827957] req@ffff9e6803e98780 x1874668790374784/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827957.955177:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790374784/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827957.955186:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827957.955189:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827957.955209:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790374784:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827958.465769:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827958.658014:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827959.489760:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827959.682032:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827960.513856:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827960.706025:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827961.537806:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827961.729898:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827962.561806:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827962.754055:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827963.074017:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827963.074034:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827963.074042:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827963.074059:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4599 00000100:00080000:3.0:1787827963.074078:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827963.074090:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827963.074162:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790374912/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827963.074206:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827963.074211:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827963.074214:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827963.074220:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4599 00000100:00080000:3.0:1787827963.074224:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827963.074229:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827963.074244:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827963.074265:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790375040/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827963.074279:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827963.074284:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827963.074287:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827963.074293:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4599 00000100:00080000:3.0:1787827963.074297:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827963.074311:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827963.074314:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790374912:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787827963.074324:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827963.074341:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790375168/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827963.074360:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827963.074366:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827963.074368:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827963.074374:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4599 00000100:00080000:3.0:1787827963.074378:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827963.074383:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827963.074392:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827963.074407:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790375296/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827963.074419:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827963.074424:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827963.074427:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827963.074437:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4599 00000800:00000100:2.0:1787827963.074445:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827963.074446:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827963.074459:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827963.074465:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827963.074480:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790375424/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827963.074496:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827963.074502:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827963.074505:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827963.074510:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4599 00000100:00080000:3.0:1787827963.074515:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827963.074522:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827963.074530:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827963.074556:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827963/real 1787827963] req@ffff9e680a2d6580 x1874668790374912/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827963.074556:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790375552/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827963.074633:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827963.074656:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827963.074659:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827963.074670:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4599 00000100:00080000:3.0:1787827963.074675:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827963.074680:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790374912/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827963.074689:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787827963.074776:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787827963.074779:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787827963.074794:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827963.074805:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790374912:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787827963.074809:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790375680/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827963.074839:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790375040:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827963.074843:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827963.074860:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827963.074887:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827963.074896:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4599 00000100:00080000:3.0:1787827963.074903:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827963.074909:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827963.074918:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827963.074935:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827963.074945:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790375808/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827963.074966:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827963.074974:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827963/real 1787827963] req@ffff9e680a2d5a40 x1874668790375040/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827963.074977:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827963.074984:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827963.075006:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4599 00000100:00080000:3.0:1787827963.075020:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827963.075030:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827963.075032:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790375040/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787827963.075044:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787827963.075060:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827963.075073:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787827963.075075:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790375936/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827963.075090:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790375040:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827963.075157:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790375168:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827963.075188:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827963.075201:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827963/real 1787827963] req@ffff9e680a2d4000 x1874668790375168/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827963.075216:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790375168/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827963.075227:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827963.075231:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827963.075235:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790375168:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827963.075242:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790375296:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827963.075261:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827963.075272:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827963/real 1787827963] req@ffff9e680a2d61c0 x1874668790375296/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827963.075286:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790375296/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827963.075296:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827963.075300:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827963.075303:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790375296:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827963.075311:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790375424:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827963.075329:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827963.075339:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827963/real 1787827963] req@ffff9e680a2d5e00 x1874668790375424/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827963.075353:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790375424/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827963.075363:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827963.075367:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827963.075371:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790375424:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827963.075377:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790375552:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827963.075396:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827963.075406:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827963/real 1787827963] req@ffff9e680a2d4780 x1874668790375552/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827963.075419:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790375552/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827963.075439:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827963.075443:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827963.075447:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790375552:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827963.075459:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790375680:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827963.075495:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827963.075517:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827963/real 1787827963] req@ffff9e680a2d43c0 x1874668790375680/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827963.075542:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790375680/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827963.075560:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827963.075566:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827963.075572:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790375680:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827963.075585:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790375808:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827963.075628:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827963.075652:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827963/real 1787827963] req@ffff9e680a2d4b40 x1874668790375808/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827963.075669:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790375808/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827963.075679:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827963.075683:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827963.075688:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790375808:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827963.075730:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790375936:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827963.075761:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827963.075776:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827963/real 1787827963] req@ffff9e680a2d4f00 x1874668790375936/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827963.075790:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790375936/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827963.075800:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827963.075804:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827963.075808:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790375936:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827963.585784:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827963.777924:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827964.609757:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827964.801874:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827965.633801:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827965.825960:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827966.657863:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827966.849985:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827967.681796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827967.873960:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827968.193869:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827968.193883:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827968.193888:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827968.193901:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4604 00000100:00080000:3.0:1787827968.193933:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827968.193945:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827968.193993:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790376064/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827968.194030:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827968.194035:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827968.194037:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827968.194042:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4604 00000100:00080000:3.0:1787827968.194047:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827968.194051:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827968.194063:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827968.194079:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790376192/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827968.194095:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827968.194103:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787827968.194107:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790376064:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827968.194108:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827968.194120:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4604 00000100:00080000:3.0:1787827968.194128:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827968.194133:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827968.194139:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827968.194155:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790376320/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827968.194167:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827968.194172:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827968.194175:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827968.194180:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4604 00000100:00080000:3.0:1787827968.194196:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827968.194201:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827968.194208:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827968.194222:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790376448/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827968.194234:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827968.194238:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827968.194241:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827968.194248:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4604 00000100:00080000:3.0:1787827968.194252:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827968.194256:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827968.194262:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827968.194272:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790376192:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827968.194282:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790376576/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827968.194300:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827968.194306:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827968.194309:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827968.194315:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4604 00000100:00080000:3.0:1787827968.194320:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827968.194326:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827968.194332:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790376320:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787827968.194337:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827968.194364:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790376704/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827968.194378:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827968.194380:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790376448:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787827968.194389:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827968.194396:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827968.194403:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4604 00000100:00080000:3.0:1787827968.194413:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827968.194419:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827968.194427:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827968.194431:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790376576:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827968.194450:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790376832/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827968.194469:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827968.194475:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790376704:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827968.194478:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827968.194487:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827968.194497:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4604 00000100:00080000:3.0:1787827968.194504:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827968.194509:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827968.194519:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827968.194531:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790376832:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787827968.194542:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790376960/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827968.194560:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827968.194566:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827968.194569:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827968.194575:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790376960:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787827968.194583:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4604 00000100:00080000:3.0:1787827968.194592:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827968.194598:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827968.194606:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827968.194624:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790377088/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827968.194670:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790377088:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827968.705806:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827968.897995:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827969.729796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827969.922022:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827970.753844:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827970.945958:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787827971.266098:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:3.0:1787827971.266156:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787827971.266179:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787827971.266244:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:3.0:1787827971.266261:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787827971.266285:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787827971.266316:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787827971.266351:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827968/real 1787827971] req@ffff9e680a2d4780 x1874668790377088/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787827971.266354:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787827971.266382:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787827971.266410:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787827971.266436:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790377088/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787827971.266443:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00080000:2.0:1787827971.266474:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827971.266487:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827971.266499:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790377088:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827971.266517:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827968/real 1787827971] req@ffff9e680a2d43c0 x1874668790376960/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827971.266544:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790376960/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827971.266559:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827971.266566:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827971.266573:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790376960:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827971.266587:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827968/real 1787827971] req@ffff9e680a2d4b40 x1874668790376832/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827971.266614:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790376832/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827971.266631:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827971.266638:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827971.266644:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790376832:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827971.266658:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827968/real 1787827971] req@ffff9e680a2d52c0 x1874668790376576/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827971.266682:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790376576/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827971.266753:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827971.266763:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827971.266772:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790376576:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827971.266789:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827968/real 1787827971] req@ffff9e680a2d4f00 x1874668790376704/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827971.266840:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790376704/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827971.266860:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827971.266868:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827971.266877:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790376704:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827971.266900:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827968/real 1787827971] req@ffff9e680a2d70c0 x1874668790376192/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827971.266927:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790376192/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827971.266947:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827971.266958:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827971.266966:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790376192:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827971.266982:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827968/real 1787827971] req@ffff9e680a2d6d00 x1874668790376320/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827971.267024:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790376320/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827971.267043:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827971.267054:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827971.267063:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790376320:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827971.267083:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827968/real 1787827971] req@ffff9e680a2d7c00 x1874668790376448/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827971.267132:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790376448/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827971.267152:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827971.267160:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827971.267169:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790376448:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827971.267181:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827968/real 1787827971] req@ffff9e680a2d7480 x1874668790376064/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827971.267206:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790376064/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827971.267223:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827971.267232:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827971.267242:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790376064:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787827971.777887:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827971.969949:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827972.801855:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827972.993969:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827973.313905:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827973.313918:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827973.313923:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827973.313935:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4610 00000100:00080000:1.0:1787827973.313948:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827973.313957:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787827973.314000:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790377216/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827973.314039:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827973.314046:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827973.314050:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827973.314058:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4610 00000100:00080000:1.0:1787827973.314066:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827973.314076:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827973.314093:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827973.314114:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790377216:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787827973.314124:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790377344/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827973.314152:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827973.314159:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827973.314164:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827973.314172:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4610 00000100:00080000:1.0:1787827973.314182:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827973.314190:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827973.314201:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827973.314221:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827973.314245:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790377472/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827973.314269:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827973.314277:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827973.314281:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827973.314290:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4610 00000100:00080000:1.0:1787827973.314300:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827973.314310:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827973/real 1787827973] req@ffff9e6831aa8f00 x1874668790377216/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827973.314318:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827973.314335:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827973.314364:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790377216/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827973.314368:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790377600/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827973.314447:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827973.314448:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827973.314466:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787827973.314473:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827973.314479:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827973.314489:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4610 00000100:00100000:2.0:1787827973.314502:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790377216:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787827973.314503:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827973.314518:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827973.314534:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827973.314561:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790377728/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827973.314567:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790377344:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827973.314600:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827973.314609:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827973.314613:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827973.314621:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4610 00000100:00080000:1.0:1787827973.314634:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827973.314642:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827973.314652:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827973.314659:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787827973.314686:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790377856/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827973.314749:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827973/real 1787827973] req@ffff9e6831aa8b40 x1874668790377344/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827973.314782:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790377344/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827973.314823:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827973.314844:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787827973.314852:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827973.314856:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827973.314865:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4610 00000100:00080000:2.0:1787827973.314874:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787827973.314881:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827973.314890:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790377344:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827973.314900:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827973.314916:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827973.314920:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790377472:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787827973.314949:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790377984/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827973.314970:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827973.314976:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827973.314980:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787827973.314990:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787827973.314994:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4610 00000100:00080000:1.0:1787827973.315007:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827973.315017:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787827973.315026:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827973.315031:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827973/real 1787827973] req@ffff9e6831aa9680 x1874668790377472/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827973.315054:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790378112/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827973.315070:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790377472/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827973.315084:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827973.315102:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827973.315104:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827973.315107:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827973.315119:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787827973.315126:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4610 00000100:00100000:2.0:1787827973.315138:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790377472:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787827973.315141:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827973.315162:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827973.315170:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790377600:192.168.203.109@tcp:8:kworker.0 00000008:00100000:1.0:1787827973.315178:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787827973.315210:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790378240/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827973.315235:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827973.315275:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827973/real 1787827973] req@ffff9e6831aaa580 x1874668790377600/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827973.315307:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790377600/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827973.315325:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827973.315332:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827973.315339:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790377600:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827973.315375:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790377728:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827973.315419:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827973.315445:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827973/real 1787827973] req@ffff9e6831aa9e00 x1874668790377728/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827973.315475:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790377728/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827973.315504:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827973.315511:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827973.315517:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790377728:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827973.315528:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790377856:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827973.315569:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827973.315593:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827973/real 1787827973] req@ffff9e6831aaa940 x1874668790377856/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827973.315619:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790377856/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827973.315638:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827973.315644:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827973.315650:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790377856:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827973.315662:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790377984:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827973.315753:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827973.315780:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827973/real 1787827973] req@ffff9e6831aa8780 x1874668790377984/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827973.315829:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790377984/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827973.315855:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827973.315861:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827973.315868:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790377984:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827973.315882:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790378112:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827973.315922:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827973.315944:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827973/real 1787827973] req@ffff9e6831aaa1c0 x1874668790378112/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827973.315967:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790378112/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827973.315983:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827973.315989:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827973.315996:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790378112:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827973.316008:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790378240:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827973.316049:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827973.316078:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827973/real 1787827973] req@ffff9e6831aa8000 x1874668790378240/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827973.316103:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790378240/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827973.316118:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827973.316124:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827973.316131:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790378240:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827973.825872:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827974.018026:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827974.849756:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827975.041861:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827975.873846:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827976.065963:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827976.897765:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827977.089901:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827977.921847:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827978.113940:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827978.433823:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827978.433834:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827978.433840:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827978.433851:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4615 00000100:00080000:3.0:1787827978.433862:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827978.433871:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827978.433910:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790378368/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827978.433943:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827978.433948:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827978.433950:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827978.433955:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4615 00000100:00080000:3.0:1787827978.433959:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827978.433963:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827978.433972:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827978.433999:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790378496/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827978.434013:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790378368:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827978.434041:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827978.434045:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827978.434048:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827978.434056:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4615 00000100:00080000:3.0:1787827978.434062:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827978.434069:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827978.434078:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827978.434101:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790378624/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827978.434107:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827978.434129:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827978.434136:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827978.434140:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827978.434147:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4615 00000100:00080000:3.0:1787827978.434154:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827978.434159:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827978/real 1787827978] req@ffff9e680a2d5e00 x1874668790378368/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827978.434166:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827978.434182:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827978.434194:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790378368/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827978.434204:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790378752/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827978.434212:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827978.434220:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827978.434225:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790378368:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787827978.434255:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790378496:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827978.434285:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827978.434303:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827978/real 1787827978] req@ffff9e680a2d61c0 x1874668790378496/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827978.434305:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827978.434317:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827978.434323:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827978.434336:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4615 00000100:00080000:3.0:1787827978.434344:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827978.434350:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827978.434356:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827978.434369:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790378496/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827978.434376:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790378880/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827978.434397:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827978.434403:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827978.434409:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790378496:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827978.434419:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790378624:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827978.434466:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827978.434488:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827978/real 1787827978] req@ffff9e680a2d4000 x1874668790378624/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827978.434514:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790378624/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827978.434562:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827978.434565:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827978.434575:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827978.434582:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827978.434583:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827978.434593:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790378624:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827978.434599:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4615 00000100:00080000:3.0:1787827978.434611:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827978.434622:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827978.434642:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790378752:192.168.203.109@tcp:8:kworker.0 00000008:00100000:3.0:1787827978.434733:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827978.434734:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787827978.434778:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790379008/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827978.434789:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827978/real 1787827978] req@ffff9e680a2d5a40 x1874668790378752/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827978.434813:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827978.434819:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827978.434822:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827978.434826:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790378752/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827978.434831:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4615 00000100:00080000:3.0:1787827978.434843:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827978.434851:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827978.434852:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827978.434864:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787827978.434867:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827978.434879:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790378752:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827978.434883:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790379136/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827978.434904:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790378880:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787827978.434911:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827978.434919:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827978.434921:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827978.434926:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4615 00000100:00080000:3.0:1787827978.434930:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827978.434933:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827978.434938:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827978.434951:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790379264/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827978.434964:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827978.434967:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827978.434972:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827978.434975:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827978.434980:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4615 00000100:00080000:3.0:1787827978.434985:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827978.434988:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827978/real 1787827978] req@ffff9e680a2d6580 x1874668790378880/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827978.434993:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827978.435004:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827978.435017:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790378880/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827978.435022:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790379392/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827978.435074:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827978.435082:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827978.435087:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790378880:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827978.435109:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790379008:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827978.435130:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827978.435140:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827978/real 1787827978] req@ffff9e680a2d4000 x1874668790379008/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827978.435152:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790379008/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827978.435162:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827978.435165:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827978.435168:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790379008:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827978.435173:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790379136:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827978.435188:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827978.435197:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827978/real 1787827978] req@ffff9e680a2d61c0 x1874668790379136/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827978.435208:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790379136/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827978.435216:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827978.435218:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827978.435221:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790379136:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827978.435226:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790379264:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827978.435241:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827978.435251:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827978/real 1787827978] req@ffff9e680a2d5e00 x1874668790379264/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827978.435261:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790379264/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827978.435270:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827978.435272:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827978.435275:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790379264:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827978.435298:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790379392:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827978.435329:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827978.435352:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827978/real 1787827978] req@ffff9e680a2d7480 x1874668790379392/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827978.435376:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790379392/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827978.435393:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827978.435400:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827978.435407:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790379392:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827978.945865:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827979.137976:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827979.969851:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827980.161987:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827980.993863:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827981.185972:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827982.017872:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827982.209966:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827983.041841:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827983.233953:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787827983.553867:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827983.553880:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787827983.553887:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827983.553899:0:51616:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4620 00000100:00080000:3.0:1787827983.553914:0:51616:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827983.553924:0:51616:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787827983.553974:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790379520/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827983.554010:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827983.554015:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787827983.554017:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827983.554022:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4620 00000100:00080000:3.0:1787827983.554027:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827983.554033:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827983.554045:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827983.554061:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790379648/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827983.554078:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827983.554086:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787827983.554089:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790379520:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787827983.554091:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827983.554106:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4620 00000100:00080000:3.0:1787827983.554115:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827983.554120:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827983.554129:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827983.554146:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790379776/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827983.554161:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827983.554165:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787827983.554169:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827983.554175:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4620 00000100:00080000:3.0:1787827983.554182:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827983.554187:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787827983.554196:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787827983.554197:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827983.554220:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790379904/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827983.554232:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827983.554236:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827983.554238:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827983.554242:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4620 00000100:00080000:3.0:1787827983.554246:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827983.554266:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827983.554273:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827983/real 1787827983] req@ffff9e680a2d7c00 x1874668790379520/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787827983.554277:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787827983.554303:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790380032/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827983.554332:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827983.554336:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787827983.554341:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790379520/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827983.554342:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827983.554359:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4620 00000100:00080000:3.0:1787827983.554369:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827983.554384:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827983.554395:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787827983.554396:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827983.554411:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790379520:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787827983.554411:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787827983.554438:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790380160/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787827983.554465:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827983.554471:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790379648:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827983.554480:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787827983.554489:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827983.554500:0:51616:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4620 00000100:00080000:3.0:1787827983.554509:0:51616:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827983.554515:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787827983.554529:0:51616:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827983.554540:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827983/real 1787827983] req@ffff9e680a2d6d00 x1874668790379648/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787827983.554543:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827983.554569:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790379648/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827983.554583:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790380288/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827983.554588:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827983.554602:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827983.554615:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790379648:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827983.554616:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827983.554628:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787827983.554634:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790379776:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787827983.554636:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827983.554652:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4620 00000100:00080000:3.0:1787827983.554660:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787827983.554666:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787827983.554675:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787827983.554676:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827983.554756:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827983/real 1787827983] req@ffff9e680a2d70c0 x1874668790379776/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787827983.554784:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790380416/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827983.554798:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790379776/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787827983.554804:0:51616:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787827983.554820:0:51616:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787827983.554827:0:51616:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787827983.554843:0:51616:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4620 00000100:00080000:2.0:1787827983.554849:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787827983.554853:0:51616:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827983.554861:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787827983.554868:0:51616:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827983.554873:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790379776:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787827983.554882:0:51616:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827983.554887:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790379904:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827983.554920:0:51616:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790380544/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827983.554935:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827983.554962:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827983/real 1787827983] req@ffff9e680a2d4f00 x1874668790379904/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827983.554979:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790379904/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827983.554996:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827983.555002:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827983.555009:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790379904:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827983.555021:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790380032:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827983.555060:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827983.555082:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827983/real 1787827983] req@ffff9e680a2d52c0 x1874668790380032/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827983.555107:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790380032/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827983.555118:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827983.555121:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827983.555126:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790380032:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827983.555162:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790380160:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827983.555190:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827983.555204:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827983/real 1787827983] req@ffff9e680a2d4b40 x1874668790380160/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827983.555228:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790380160/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827983.555240:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827983.555244:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827983.555247:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790380160:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827983.555254:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790380288:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827983.555273:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827983.555284:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827983/real 1787827983] req@ffff9e680a2d43c0 x1874668790380288/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827983.555298:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790380288/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827983.555307:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827983.555310:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827983.555314:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790380288:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827983.555320:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790380416:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827983.555338:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827983.555350:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827983/real 1787827983] req@ffff9e680a2d4780 x1874668790380416/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827983.555364:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790380416/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827983.555385:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827983.555388:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827983.555392:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790380416:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827983.555398:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790380544:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827983.555416:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827983.555427:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827983/real 1787827983] req@ffff9e680a2d7c00 x1874668790380544/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827983.555440:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790380544/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827983.555449:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827983.555453:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827983.555456:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790380544:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827984.065866:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827984.257973:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827985.089818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827985.281939:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827986.113821:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827986.305996:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827987.137811:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827987.329931:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827988.161879:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827988.353824:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827988.673812:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827988.673823:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827988.673827:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827988.673839:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4625 00000100:00080000:0.0:1787827988.673851:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827988.673859:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827988.673901:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790380672/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827988.673936:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827988.673940:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827988.673943:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827988.673947:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4625 00000100:00080000:0.0:1787827988.673950:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827988.673954:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827988.673966:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827988.674004:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790380800/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827988.674013:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827988.674018:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787827988.674018:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790380672:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787827988.674021:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827988.674028:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4625 00000100:00080000:0.0:1787827988.674031:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827988.674034:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827988.674037:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827988.674045:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790380928/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827988.674051:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827988.674054:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827988.674056:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827988.674059:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4625 00000100:00080000:0.0:1787827988.674062:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827988.674064:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827988.674067:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827988.674076:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790381056/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827988.674082:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827988.674089:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827988.674091:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827988.674093:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827988.674096:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4625 00000100:00080000:0.0:1787827988.674100:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827988.674103:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827988.674106:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827988.674114:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790381184/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827988.674120:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827988.674123:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827988.674124:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827988.674127:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4625 00000100:00080000:0.0:1787827988.674129:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827988.674131:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827988/real 1787827988] req@ffff9e6803e9ad00 x1874668790380672/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827988.674135:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827988.674141:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827988.674155:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790381312/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827988.674159:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790380672/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827988.674166:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827988.674173:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827988.674176:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827988.674182:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4625 00000100:00080000:2.0:1787827988.674183:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827988.674186:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827988.674190:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787827988.674193:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827988.674198:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827988.674198:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790380672:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827988.674209:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790381440/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827988.674215:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827988.674217:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827988.674219:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827988.674222:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4625 00000100:00100000:2.0:1787827988.674225:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790380800:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827988.674227:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827988.674234:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827988.674238:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827988.674246:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790381568/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827988.674254:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827988.674255:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827988.674260:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827988.674262:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827988.674266:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4625 00000100:00080000:0.0:1787827988.674269:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827988.674272:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827988/real 1787827988] req@ffff9e6803e98b40 x1874668790380800/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827988.674275:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827988.674281:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827988.674288:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790380800/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827988.674292:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790381696/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827988.674311:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827988.674316:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827988.674320:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790380800:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827988.674325:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790380928:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827988.674339:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827988.674345:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827988/real 1787827988] req@ffff9e6803e9bc00 x1874668790380928/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827988.674376:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790380928/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827988.674386:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827988.674390:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827988.674393:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790380928:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827988.674401:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790381056:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827988.674437:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827988.674449:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827988/real 1787827988] req@ffff9e6803e992c0 x1874668790381056/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827988.674478:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790381056/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827988.674488:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827988.674492:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827988.674495:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790381056:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827988.674501:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790381184:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827988.674520:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827988.674529:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827988/real 1787827988] req@ffff9e6803e98780 x1874668790381184/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827988.674541:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790381184/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827988.674549:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827988.674552:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827988.674555:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790381184:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827988.674561:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790381312:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827988.674576:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827988.674585:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827988/real 1787827988] req@ffff9e6803e9b480 x1874668790381312/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827988.674598:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790381312/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827988.674607:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827988.674610:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827988.674613:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790381312:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827988.674634:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790381440:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827988.674647:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827988.674653:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827988/real 1787827988] req@ffff9e6803e9a1c0 x1874668790381440/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827988.674660:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790381440/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827988.674665:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827988.674667:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827988.674669:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790381440:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827988.674681:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790381568:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827988.674732:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827988.674741:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827988/real 1787827988] req@ffff9e6803e9a580 x1874668790381568/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827988.674749:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790381568/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827988.674760:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827988.674762:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827988.674764:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790381568:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827988.674767:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790381696:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827988.674806:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827988.674818:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827988/real 1787827988] req@ffff9e6803e9a940 x1874668790381696/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827988.674830:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790381696/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827988.674839:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827988.674842:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827988.674845:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790381696:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827989.185821:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827989.377948:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827990.209795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827990.401907:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827991.233806:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827991.426344:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827992.257860:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827992.450021:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827993.281837:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827993.473950:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827993.793811:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827993.793823:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827993.793828:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827993.793841:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4630 00000100:00080000:0.0:1787827993.793855:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827993.793864:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827993.793923:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790381824/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827993.793982:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827993.793986:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827993.793988:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827993.794002:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4630 00000100:00080000:0.0:1787827993.794006:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827993.794010:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827993.794021:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827993.794034:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790381952/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827993.794051:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827993.794055:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827993.794056:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827993.794060:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4630 00000100:00080000:0.0:1787827993.794063:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827993.794067:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827993.794071:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827993.794081:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790382080/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827993.794099:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827993.794102:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827993.794104:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827993.794111:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4630 00000100:00080000:0.0:1787827993.794115:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827993.794128:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787827993.794128:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790381824:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1787827993.794143:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827993.794166:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790382208/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827993.794182:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827993.794188:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827993.794192:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827993.794198:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4630 00000100:00080000:0.0:1787827993.794205:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827993.794225:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827993.794226:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827993.794239:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827993.794279:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790382336/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827993.794310:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827993.794317:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827993.794326:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827993.794328:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827993/real 1787827993] req@ffff9e6803e99680 x1874668790381824/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827993.794348:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4630 00000100:00080000:0.0:1787827993.794363:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827993.794370:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790381824/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827993.794378:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827993.794410:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787827993.794411:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827993.794425:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827993.794440:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790381824:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827993.794455:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790382464/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827993.794457:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790381952:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827993.794486:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827993.794491:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827993.794495:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827993.794518:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4630 00000800:00000100:2.0:1787827993.794519:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827993.794529:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827993.794538:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827993.794547:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827993.794548:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827993/real 1787827993] req@ffff9e6803e99e00 x1874668790381952/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827993.794567:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790382592/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827993.794575:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790381952/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827993.794601:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827993.794612:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827993.794613:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827993.794617:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827993.794624:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787827993.794641:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4630 00000100:00100000:2.0:1787827993.794642:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790381952:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827993.794696:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827993.794706:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827993.794779:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827993.794797:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790382080:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787827993.794813:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790382720/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827993.794843:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787827993.794868:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827993.794869:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827993/real 1787827993] req@ffff9e6803e98f00 x1874668790382080/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827993.794895:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827993.794904:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827993.794920:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790382080/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827993.794922:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4630 00000100:00080000:0.0:1787827993.794953:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827993.794957:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827993.794965:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787827993.794968:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827993.794979:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790382080:192.168.203.109@tcp:38:kworker.0 00000008:00100000:0.0:1787827993.794982:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827993.794990:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790382208:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787827993.795009:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790382848/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827993.795031:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827993.795055:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827993/real 1787827993] req@ffff9e6803e9b840 x1874668790382208/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827993.795077:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790382208/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827993.795102:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827993.795106:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827993.795109:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790382208:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827993.795116:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790382336:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827993.795140:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827993.795152:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827993/real 1787827993] req@ffff9e6803e9a940 x1874668790382336/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827993.795164:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790382336/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827993.795172:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827993.795175:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827993.795178:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790382336:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827993.795183:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790382464:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827993.795198:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827993.795207:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827993/real 1787827993] req@ffff9e6803e9a580 x1874668790382464/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827993.795228:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790382464/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827993.795237:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827993.795257:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827993.795260:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790382464:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827993.795268:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790382592:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827993.795302:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827993.795314:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827993/real 1787827993] req@ffff9e6803e9a1c0 x1874668790382592/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827993.795327:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790382592/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827993.795337:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827993.795340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827993.795344:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790382592:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827993.795367:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790382720:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827993.795387:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827993.795398:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827993/real 1787827993] req@ffff9e6803e9b480 x1874668790382720/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827993.795411:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790382720/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827993.795438:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827993.795442:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827993.795446:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790382720:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827993.795452:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790382848:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827993.795471:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827993.795481:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827993/real 1787827993] req@ffff9e6803e98780 x1874668790382848/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827993.795494:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790382848/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827993.795514:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827993.795518:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827993.795521:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790382848:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827994.305832:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827994.498000:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827995.329830:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827995.521961:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827996.353877:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827996.546077:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827997.377902:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827997.569930:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787827998.401826:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827998.594029:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787827998.913809:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827998.913832:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787827998.913837:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827998.913848:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4635 00000100:00080000:0.0:1787827998.913862:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827998.913871:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787827998.913926:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790382976/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827998.913970:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827998.913974:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827998.913976:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827998.913979:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4635 00000100:00080000:0.0:1787827998.913982:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827998.913986:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827998.913995:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827998.914006:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790383104/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827998.914015:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827998.914018:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827998.914020:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827998.914022:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4635 00000100:00080000:0.0:1787827998.914025:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827998.914027:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827998.914030:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827998.914037:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790383232/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827998.914045:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827998.914048:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827998.914049:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827998.914052:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4635 00000100:00080000:0.0:1787827998.914055:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827998.914058:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827998.914073:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827998.914080:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790382976:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787827998.914084:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790383360/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827998.914096:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827998.914099:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787827998.914102:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827998.914106:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4635 00000100:00080000:0.0:1787827998.914108:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827998.914111:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827998.914114:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827998.914127:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790383488/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827998.914135:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827998.914138:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787827998.914140:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827998.914143:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4635 00000100:00080000:0.0:1787827998.914146:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827998.914149:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827998.914153:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827998.914160:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787827998.914167:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790383616/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827998.914176:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827998.914179:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787827998.914180:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827998.914184:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4635 00000100:00080000:0.0:1787827998.914189:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827998.914192:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827998.914197:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787827998.914207:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790383744/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787827998.914217:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827998.914221:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827998/real 1787827998] req@ffff9e6803e992c0 x1874668790382976/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787827998.914222:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787827998.914226:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787827998.914253:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4635 00000100:00080000:0.0:1787827998.914261:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827998.914271:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827998.914280:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787827998.914288:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790382976/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787827998.914297:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790383872/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827998.914315:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787827998.914318:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787827998.914326:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827998.914327:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787827998.914331:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827998.914339:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790382976:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787827998.914343:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4635 00000100:00100000:2.0:1787827998.914362:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790383104:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787827998.914383:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787827998.914391:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787827998.914396:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787827998.914408:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790384000/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827998.914425:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827998.914447:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827998/real 1787827998] req@ffff9e6803e9bc00 x1874668790383104/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827998.914460:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790383104/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827998.914468:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827998.914471:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827998.914474:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790383104:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827998.914479:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790383232:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827998.914493:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827998.914501:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827998/real 1787827998] req@ffff9e6803e98b40 x1874668790383232/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827998.914510:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790383232/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827998.914516:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827998.914519:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827998.914521:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790383232:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827998.914543:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790383360:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827998.914557:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827998.914565:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827998/real 1787827998] req@ffff9e6803e9ad00 x1874668790383360/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827998.914575:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790383360/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827998.914582:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827998.914585:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827998.914587:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790383360:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827998.914591:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790383488:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827998.914604:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827998.914611:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827998/real 1787827998] req@ffff9e6803e98780 x1874668790383488/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827998.914644:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790383488/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827998.914652:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827998.914654:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827998.914657:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790383488:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827998.914662:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790383616:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827998.914690:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827998.914727:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827998/real 1787827998] req@ffff9e6803e9b480 x1874668790383616/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827998.914737:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790383616/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827998.914751:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827998.914754:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827998.914756:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790383616:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827998.914761:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790383744:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787827998.914778:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827998.914787:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827998/real 1787827998] req@ffff9e6803e9a1c0 x1874668790383744/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827998.914796:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790383744/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827998.914803:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827998.914805:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827998.914807:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790383744:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787827998.914812:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790383872:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827998.914825:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827998.914833:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827998/real 1787827998] req@ffff9e6803e9a580 x1874668790383872/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827998.914842:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790383872/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827998.914848:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827998.914850:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827998.914853:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790383872:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787827998.914858:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790384000:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787827998.914870:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787827998.914877:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827998/real 1787827998] req@ffff9e6803e9a940 x1874668790384000/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827998.914885:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790384000/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827998.914891:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827998.914894:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787827998.914896:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790384000:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787827999.425784:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787827999.617957:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828000.449823:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828000.641836:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828001.473784:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828001.665954:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828002.497837:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828002.690009:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828003.521768:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828003.713880:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828004.033806:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828004.033816:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828004.033821:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828004.033831:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4640 00000100:00080000:1.0:1787828004.033843:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828004.033851:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828004.033894:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790384128/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828004.033918:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828004.033927:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828004.033928:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828004.033931:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4640 00000100:00080000:1.0:1787828004.033934:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828004.033937:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828004.033947:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828004.033958:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790384256/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828004.033966:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828004.033968:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828004.033970:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828004.033974:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4640 00000100:00080000:1.0:1787828004.033976:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828004.033979:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828004.033989:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790384128:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787828004.033993:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828004.034014:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790384384/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828004.034027:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828004.034031:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828004.034033:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828004.034037:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4640 00000100:00080000:1.0:1787828004.034041:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828004.034046:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828004.034053:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828004.034067:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790384512/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828004.034077:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828004.034082:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828004.034085:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828004.034091:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828004.034100:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4640 00000100:00080000:1.0:1787828004.034106:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828004.034111:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828004.034118:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828004.034131:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790384640/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828004.034139:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828004.034141:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828004.034144:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828004.034148:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828004/real 1787828004] req@ffff9e6831aab480 x1874668790384128/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828004.034149:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4640 00000100:00080000:1.0:1787828004.034157:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828004.034166:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828004.034172:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828004.034186:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790384768/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828004.034194:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828004.034201:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828004.034201:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790384128/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828004.034203:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828004.034212:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4640 00000100:00080000:1.0:1787828004.034221:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828004.034229:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828004.034238:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828004.034246:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828004.034249:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790384896/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828004.034258:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828004.034263:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828004.034269:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828004.034272:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828004.034272:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790384128:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828004.034286:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4640 00000100:00080000:1.0:1787828004.034294:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828004.034298:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828004.034305:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828004.034305:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790384256:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828004.034319:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790385024/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828004.034327:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828004.034329:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828004.034330:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828004.034333:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4640 00000100:00080000:1.0:1787828004.034336:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828004.034341:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828004.034348:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828004.034361:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828004.034366:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790385152/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828004.034383:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828004/real 1787828004] req@ffff9e6831aabc00 x1874668790384256/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828004.034404:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790384256/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828004.034415:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828004.034419:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828004.034423:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790384256:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828004.034450:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790384384:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828004.034476:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828004.034490:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828004/real 1787828004] req@ffff9e6831aaad00 x1874668790384384/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828004.034502:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790384384/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828004.034510:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828004.034514:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828004.034517:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790384384:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828004.034524:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790384512:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828004.034546:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828004.034558:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828004/real 1787828004] req@ffff9e6831aa9a40 x1874668790384512/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828004.034570:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790384512/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828004.034579:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828004.034599:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828004.034602:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790384512:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828004.034610:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790384640:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828004.034633:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828004.034646:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828004/real 1787828004] req@ffff9e6831aa8000 x1874668790384640/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828004.034658:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790384640/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828004.034666:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828004.034670:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828004.034673:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790384640:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828004.034680:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790384768:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828004.034734:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828004.034747:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828004/real 1787828004] req@ffff9e6831aaa1c0 x1874668790384768/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828004.034760:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790384768/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828004.034770:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828004.034773:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828004.034778:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790384768:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828004.034784:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790384896:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828004.034805:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828004.034817:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828004/real 1787828004] req@ffff9e6831aa8780 x1874668790384896/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828004.034830:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790384896/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828004.034838:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828004.034841:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828004.034845:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790384896:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828004.034852:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790385024:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828004.034871:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828004.034883:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828004/real 1787828004] req@ffff9e6831aaa940 x1874668790385024/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828004.034895:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790385024/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828004.034903:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828004.034907:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828004.034910:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790385024:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828004.034917:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790385152:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828004.034938:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828004.034949:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828004/real 1787828004] req@ffff9e6831aa9e00 x1874668790385152/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828004.034962:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790385152/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828004.034971:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828004.034974:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828004.034978:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790385152:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828004.545757:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828004.737825:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828005.569789:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828005.761877:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828006.593800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828006.785939:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828007.617832:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828007.809922:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828008.641797:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828008.833928:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828009.153871:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828009.153887:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828009.153896:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828009.153912:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4645 00000100:00080000:3.0:1787828009.153929:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828009.153939:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828009.153986:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790385280/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828009.154023:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828009.154028:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828009.154031:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828009.154037:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4645 00000100:00080000:3.0:1787828009.154041:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828009.154046:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828009.154073:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828009.154096:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790385280:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828009.154105:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790385408/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828009.154132:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828009.154140:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828009.154145:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828009.154153:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4645 00000100:00080000:3.0:1787828009.154178:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828009.154188:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828009.154192:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828009.154203:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828009.154225:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790385536/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828009.154249:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828009/real 1787828009] req@ffff9e680a2d7480 x1874668790385280/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828009.154253:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828009.154267:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828009.154275:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828009.154289:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4645 00000100:00080000:3.0:1787828009.154295:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828009.154299:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790385280/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828009.154306:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828009.154319:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828009.154328:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828009.154339:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828009.154347:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790385664/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828009.154350:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790385280:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828009.154378:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828009.154386:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790385408:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828009.154387:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828009.154395:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828009.154405:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4645 00000100:00080000:3.0:1787828009.154412:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828009.154420:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828009.154428:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828009.154433:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828009.154454:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790385792/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828009.154458:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828009/real 1787828009] req@ffff9e680a2d5e00 x1874668790385408/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828009.154501:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828009.154510:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787828009.154511:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790385408/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828009.154514:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828009.154531:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4645 00000100:00080000:3.0:1787828009.154543:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828009.154577:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787828009.154595:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828009.154601:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828009.154603:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828009.154615:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790385408:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828009.154628:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790385920/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828009.154630:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790385536:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828009.154656:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828009.154662:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828009.154665:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828009.154672:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4645 00000100:00080000:3.0:1787828009.154679:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828009.154682:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828009.154690:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828009.154754:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828009.154755:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828009/real 1787828009] req@ffff9e680a2d61c0 x1874668790385536/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828009.154783:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790386048/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828009.154786:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790385536/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828009.154841:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828009.154842:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828009.154856:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828009.154858:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828009.154861:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828009.154876:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4645 00000100:00100000:2.0:1787828009.154878:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790385536:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828009.154882:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828009.154894:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828009.154902:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828009.154907:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790385664:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828009.154920:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790386176/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828009.154944:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828009.154949:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828009.154952:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828009.154959:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4645 00000800:00000100:2.0:1787828009.154969:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828009.154970:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828009.154982:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828009.154990:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828009.155005:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828009/real 1787828009] req@ffff9e680a2d4000 x1874668790385664/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828009.155013:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790386304/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828009.155054:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790385664/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828009.155068:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828009.155074:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828009.155080:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790385664:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828009.155106:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790385792:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828009.155132:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828009.155145:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828009/real 1787828009] req@ffff9e680a2d6580 x1874668790385792/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828009.155159:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790385792/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828009.155170:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828009.155173:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828009.155177:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790385792:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828009.155183:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790385920:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828009.155201:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828009.155212:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828009/real 1787828009] req@ffff9e680a2d5a40 x1874668790385920/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828009.155226:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790385920/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828009.155237:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828009.155240:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828009.155243:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790385920:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828009.155250:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790386048:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828009.155268:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828009.155279:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828009/real 1787828009] req@ffff9e680a2d7480 x1874668790386048/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828009.155293:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790386048/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828009.155302:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828009.155305:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828009.155309:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790386048:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828009.155315:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790386176:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828009.155333:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828009.155343:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828009/real 1787828009] req@ffff9e680a2d7c00 x1874668790386176/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828009.155356:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790386176/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828009.155366:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828009.155369:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828009.155373:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790386176:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828009.155379:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790386304:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828009.155398:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828009.155408:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828009/real 1787828009] req@ffff9e680a2d4780 x1874668790386304/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828009.155421:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790386304/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828009.155431:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828009.155435:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828009.155439:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790386304:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828009.665830:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828009.857956:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828010.689880:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828010.881954:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828011.713821:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828011.905947:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828012.737862:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828012.929990:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828013.761824:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828013.953957:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828014.273880:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828014.273893:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828014.273899:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828014.273915:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4650 00000100:00080000:1.0:1787828014.273940:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828014.273950:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828014.274005:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790386432/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828014.274057:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828014.274066:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828014.274071:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828014.274079:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4650 00000100:00080000:1.0:1787828014.274090:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828014.274099:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828014.274113:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828014.274139:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790386432:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828014.274147:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790386560/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828014.274166:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828014.274180:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828014.274183:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828014.274189:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4650 00000100:00080000:1.0:1787828014.274195:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828014.274202:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828014.274211:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828014.274232:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790386688/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828014.274249:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828014.274262:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828014.274274:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828014.274280:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828014.274292:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4650 00000100:00080000:1.0:1787828014.274304:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828014.274333:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828014.274345:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828014/real 1787828014] req@ffff9e6831aaa580 x1874668790386432/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828014.274346:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828014.274376:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790386816/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828014.274399:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828014.274405:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787828014.274412:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790386432/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828014.274413:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828014.274430:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4650 00000100:00080000:1.0:1787828014.274442:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828014.274444:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828014.274453:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787828014.274460:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:1.0:1787828014.274465:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828014.274470:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790386432:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828014.274487:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790386944/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828014.274506:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828014.274539:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790386560:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828014.274540:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828014.274548:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828014.274574:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4650 00000100:00080000:1.0:1787828014.274590:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828014.274602:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828014.274602:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828014.274617:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828014.274634:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828014/real 1787828014] req@ffff9e6831aa9680 x1874668790386560/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828014.274642:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790387072/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828014.274676:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828014.274679:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790386560/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828014.274687:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828014.274763:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828014.274769:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828014.274774:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4650 00000100:00080000:0.0:1787828014.274782:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828014.274784:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828014.274793:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790386560:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828014.274796:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828014.274823:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828014.274832:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790386688:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828014.274847:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790387200/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828014.274866:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828014.274871:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828014.274875:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828014.274882:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4650 00000800:00000100:0.0:1787828014.274898:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828014.274903:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828014.274913:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828014.274922:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828014.274944:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828014/real 1787828014] req@ffff9e6831aa8b40 x1874668790386688/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828014.274945:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790387328/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828014.275011:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828014.275020:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787828014.275027:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790386688/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828014.275028:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828014.275047:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4650 00000100:00080000:1.0:1787828014.275062:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828014.275069:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828014.275087:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787828014.275088:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828014.275102:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790386688:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787828014.275104:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828014.275119:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790386816:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828014.275132:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790387456/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828014.275165:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828014.275193:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828014/real 1787828014] req@ffff9e6831aa8f00 x1874668790386816/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828014.275215:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790386816/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828014.275228:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828014.275233:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828014.275238:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790386816:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828014.275261:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790386944:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828014.275282:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828014.275294:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828014/real 1787828014] req@ffff9e6831aa9e00 x1874668790386944/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828014.275314:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790386944/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828014.275330:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828014.275336:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828014.275344:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790386944:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828014.275355:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790387072:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828014.275394:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828014.275415:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828014/real 1787828014] req@ffff9e6831aaa940 x1874668790387072/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828014.275430:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790387072/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828014.275443:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828014.275447:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828014.275451:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790387072:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828014.275458:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790387200:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828014.275481:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828014.275493:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828014/real 1787828014] req@ffff9e6831aa8780 x1874668790387200/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828014.275507:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790387200/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828014.275546:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828014.275551:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828014.275555:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790387200:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828014.275563:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790387328:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828014.275589:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828014.275603:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828014/real 1787828014] req@ffff9e6831aaa1c0 x1874668790387328/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828014.275618:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790387328/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828014.275628:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828014.275631:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828014.275635:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790387328:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828014.275641:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790387456:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828014.275660:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828014.275670:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828014/real 1787828014] req@ffff9e6831aa8000 x1874668790387456/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828014.275684:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790387456/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828014.275694:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828014.275726:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828014.275730:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790387456:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828014.785766:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828014.977989:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828015.809822:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828016.001966:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828016.833780:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828017.025864:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828017.857742:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828018.049841:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828018.881847:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828019.073950:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828019.393777:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828019.393788:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828019.393791:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828019.393799:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4656 00000100:00080000:3.0:1787828019.393808:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828019.393814:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828019.393845:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790387584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828019.393868:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828019.393871:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828019.393872:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828019.393875:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4656 00000100:00080000:3.0:1787828019.393878:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828019.393880:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828019.393888:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828019.393897:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790387712/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828019.393905:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828019.393909:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787828019.393911:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790387584:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828019.393911:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828019.393919:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4656 00000100:00080000:3.0:1787828019.393921:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828019.393923:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828019.393927:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828019.393934:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790387840/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828019.393940:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828019.393942:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828019.393943:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828019.393946:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4656 00000100:00080000:3.0:1787828019.393949:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828019.393952:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828019.393955:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828019.393962:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790387968/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828019.393967:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787828019.393972:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828019.393972:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828019.393974:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828019.393978:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4656 00000100:00080000:3.0:1787828019.393981:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828019.393983:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828019.393987:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828019.393995:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790388096/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828019.394001:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828019.394004:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828019.394005:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828019.394010:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828019/real 1787828019] req@ffff9e680a2d43c0 x1874668790387584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828019.394010:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4656 00000100:00080000:3.0:1787828019.394016:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828019.394022:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828019.394027:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828019.394034:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790387584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828019.394036:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790388224/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828019.394061:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828019.394062:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828019.394065:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828019.394067:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828019.394070:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828019.394073:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4656 00000100:00080000:3.0:1787828019.394076:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828019.394077:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790387584:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828019.394082:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828019.394086:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828019.394093:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790388352/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828019.394101:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790387712:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828019.394102:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828019.394109:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828019.394112:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828019.394115:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4656 00000100:00080000:3.0:1787828019.394117:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828019.394120:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828019.394123:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828019.394128:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828019.394134:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790388480/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828019.394142:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828019/real 1787828019] req@ffff9e680a2d4b40 x1874668790387712/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828019.394142:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828019.394149:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828019.394151:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828019.394154:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4656 00000100:00080000:3.0:1787828019.394156:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828019.394162:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828019.394168:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828019.394177:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790387712/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828019.394178:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790388608/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828019.394207:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828019.394212:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828019.394215:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790387712:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828019.394219:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790387840:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828019.394233:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828019.394239:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828019/real 1787828019] req@ffff9e680a2d52c0 x1874668790387840/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828019.394246:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790387840/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828019.394252:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828019.394253:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828019.394255:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790387840:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828019.394259:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790387968:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828019.394269:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828019.394275:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828019/real 1787828019] req@ffff9e680a2d4f00 x1874668790387968/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828019.394282:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790387968/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828019.394288:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828019.394289:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828019.394291:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790387968:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828019.394294:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790388096:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828019.394303:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828019.394309:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828019/real 1787828019] req@ffff9e680a2d70c0 x1874668790388096/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828019.394316:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790388096/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828019.394321:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828019.394323:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828019.394325:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790388096:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828019.394328:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790388224:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828019.394337:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828019.394343:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828019/real 1787828019] req@ffff9e680a2d6d00 x1874668790388224/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828019.394350:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790388224/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828019.394356:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828019.394358:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828019.394359:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790388224:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828019.394373:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790388352:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828019.394383:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828019.394389:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828019/real 1787828019] req@ffff9e680a2d4780 x1874668790388352/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828019.394396:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790388352/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828019.394401:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828019.394403:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828019.394404:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790388352:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828019.394408:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790388480:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828019.394417:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828019.394422:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828019/real 1787828019] req@ffff9e680a2d7c00 x1874668790388480/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828019.394429:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790388480/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828019.394439:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828019.394441:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828019.394442:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790388480:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828019.394445:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790388608:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828019.394454:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828019.394460:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828019/real 1787828019] req@ffff9e680a2d7480 x1874668790388608/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828019.394466:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790388608/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828019.394470:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828019.394472:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828019.394488:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790388608:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828019.905756:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828020.097846:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828020.929864:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828021.121984:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828021.953835:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828022.145935:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828022.977818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828023.169879:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828024.001812:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828024.194166:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828024.513868:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828024.513880:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828024.513885:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828024.513895:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4661 00000100:00080000:0.0:1787828024.513908:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828024.513918:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828024.513995:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790388736/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828024.514039:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828024.514044:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828024.514047:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828024.514052:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4661 00000100:00080000:0.0:1787828024.514056:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828024.514061:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828024.514072:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828024.514098:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790388864/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828024.514111:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828024.514116:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828024.514118:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828024.514124:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4661 00000100:00080000:0.0:1787828024.514128:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828024.514132:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828024.514136:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828024.514148:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790388992/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828024.514156:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790388736:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828024.514173:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828024.514177:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828024.514179:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828024.514183:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4661 00000100:00080000:0.0:1787828024.514189:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828024.514195:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828024.514201:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828024.514213:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790389120/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828024.514223:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828024.514227:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828024.514228:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828024.514233:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4661 00000100:00080000:0.0:1787828024.514251:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828024.514255:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828024.514261:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828024.514267:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828024.514287:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790389248/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828024.514299:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828024.514303:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828024.514305:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828024.514309:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4661 00000100:00080000:0.0:1787828024.514312:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828024.514317:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828024.514323:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828024.514337:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828024/real 1787828024] req@ffff9e6803e9b840 x1874668790388736/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828024.514339:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790389376/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828024.514387:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828024.514401:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828024.514404:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828024.514409:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4661 00000100:00100000:1.0:1787828024.514415:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790388736/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828024.514417:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828024.514426:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828024.514436:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787828024.514465:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828024.514474:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790389504/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828024.514475:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828024.514510:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790388736:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828024.514529:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828024.514530:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790388864:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828024.514542:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828024.514547:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828024.514553:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4661 00000100:00080000:0.0:1787828024.514558:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828024.514563:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828024.514569:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828024.514577:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828024.514588:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790389632/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828024.514601:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828024/real 1787828024] req@ffff9e6803e98f00 x1874668790388864/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828024.514604:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828024.514616:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828024.514621:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828024.514634:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4661 00000100:00100000:1.0:1787828024.514640:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790388864/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828024.514641:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828024.514653:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828024.514664:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828024.514688:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828024.514701:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828024.514707:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790389760/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828024.514808:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790388864:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828024.514857:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790388992:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828024.514889:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828024.514905:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828024/real 1787828024] req@ffff9e6803e99e00 x1874668790388992/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828024.514941:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790388992/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828024.514966:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828024.514972:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828024.514993:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790388992:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828024.515002:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790389120:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828024.515032:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828024.515051:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828024/real 1787828024] req@ffff9e6803e99680 x1874668790389120/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828024.515069:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790389120/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828024.515079:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828024.515082:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828024.515085:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790389120:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828024.515092:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790389248:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828024.515114:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828024.515125:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828024/real 1787828024] req@ffff9e6803e9a940 x1874668790389248/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828024.515136:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790389248/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828024.515144:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828024.515146:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828024.515150:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790389248:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828024.515156:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790389376:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828024.515181:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828024.515190:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828024/real 1787828024] req@ffff9e6803e9a580 x1874668790389376/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828024.515201:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790389376/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828024.515210:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828024.515213:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828024.515215:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790389376:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828024.515220:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790389504:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828024.515234:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828024.515244:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828024/real 1787828024] req@ffff9e6803e9a1c0 x1874668790389504/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828024.515279:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790389504/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828024.515292:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828024.515297:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828024.515303:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790389504:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828024.515313:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790389632:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828024.515346:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828024.515380:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828024/real 1787828024] req@ffff9e6803e9b480 x1874668790389632/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828024.515406:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790389632/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828024.515416:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828024.515420:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828024.515424:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790389632:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828024.515431:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790389760:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828024.515509:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828024.515521:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828024/real 1787828024] req@ffff9e6803e98780 x1874668790389760/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828024.515553:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790389760/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828024.515563:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828024.515567:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828024.515571:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790389760:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828025.025842:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828025.217848:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828026.049764:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828026.241993:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828027.073861:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828027.266015:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828028.097757:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828028.289938:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828029.121846:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828029.314006:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828029.633784:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828029.633796:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828029.633801:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828029.633814:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4666 00000100:00080000:0.0:1787828029.633828:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828029.633837:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828029.633884:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790389888/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828029.633925:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828029.633930:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828029.633956:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828029.633961:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4666 00000100:00080000:0.0:1787828029.633966:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828029.633971:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828029.633989:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828029.634019:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790390016/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828029.634038:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828029.634038:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790389888:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828029.634053:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828029.634059:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828029.634065:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4666 00000100:00080000:0.0:1787828029.634069:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828029.634075:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828029.634087:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828029.634106:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790390144/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828029.634118:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828029.634123:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828029.634126:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828029.634130:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4666 00000100:00080000:0.0:1787828029.634135:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828029.634144:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1787828029.634152:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787828029.634154:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828029.634173:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790390272/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828029.634187:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828029.634192:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828029.634195:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828029.634198:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4666 00000100:00080000:0.0:1787828029.634203:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828029.634207:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828029.634214:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828029.634229:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790390400/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828029.634244:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828029/real 1787828029] req@ffff9e6803e9ad00 x1874668790389888/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828029.634246:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828029.634258:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828029.634263:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828029.634274:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4666 00000100:00080000:0.0:1787828029.634292:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828029.634301:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828029.634306:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828029.634323:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790390528/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828029.634328:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790389888/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828029.634338:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828029.634342:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828029.634344:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828029.634349:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4666 00000100:00080000:0.0:1787828029.634367:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828029.634377:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828029.634386:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828029.634427:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790390656/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828029.634432:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828029.634451:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787828029.634452:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828029.634462:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787828029.634464:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790389888:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828029.634467:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828029.634484:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4666 00000100:00080000:0.0:1787828029.634489:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828029.634494:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828029.634501:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828029.634515:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790390784/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828029.634523:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790390016:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828029.634540:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828029.634547:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828029.634549:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828029.634555:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4666 00000100:00080000:0.0:1787828029.634560:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828029.634563:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828029.634569:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828029.634584:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790390912/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828029.634604:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828029.634654:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828029/real 1787828029] req@ffff9e6803e98b40 x1874668790390016/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828029.634674:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790390016/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828029.634686:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828029.634691:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828029.634739:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790390016:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828029.634750:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790390144:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828029.634782:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828029.634799:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828029/real 1787828029] req@ffff9e6803e9bc00 x1874668790390144/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828029.634816:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790390144/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828029.634833:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828029.634840:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828029.634847:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790390144:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828029.634860:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790390272:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828029.634900:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828029.634923:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828029/real 1787828029] req@ffff9e6803e992c0 x1874668790390272/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828029.634942:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790390272/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828029.634953:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828029.634958:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828029.634962:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790390272:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828029.634970:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790390400:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828029.634996:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828029.635020:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828029/real 1787828029] req@ffff9e6803e98780 x1874668790390400/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828029.635034:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790390400/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828029.635043:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828029.635047:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828029.635050:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790390400:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828029.635057:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790390528:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828029.635076:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828029.635087:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828029/real 1787828029] req@ffff9e6803e9b480 x1874668790390528/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828029.635101:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790390528/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828029.635112:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828029.635116:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828029.635119:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790390528:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828029.635126:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790390656:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828029.635144:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828029.635156:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828029/real 1787828029] req@ffff9e6803e9a1c0 x1874668790390656/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828029.635170:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790390656/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828029.635179:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828029.635182:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828029.635186:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790390656:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828029.635211:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790390784:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828029.635232:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828029.635243:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828029/real 1787828029] req@ffff9e6803e9a580 x1874668790390784/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828029.635256:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790390784/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828029.635274:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828029.635277:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828029.635281:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790390784:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828029.635287:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790390912:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828029.635306:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828029.635317:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828029/real 1787828029] req@ffff9e6803e9a940 x1874668790390912/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828029.635330:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790390912/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828029.635339:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828029.635342:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828029.635345:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790390912:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828030.145877:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828030.337937:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828031.169824:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828031.361967:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828032.193832:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828032.385942:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828033.217832:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828033.409923:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828034.241755:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828034.433902:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828034.753881:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828034.753896:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828034.753903:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828034.753918:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4671 00000100:00080000:3.0:1787828034.753935:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828034.753946:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828034.754006:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790391040/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828034.754047:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828034.754052:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828034.754055:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828034.754060:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4671 00000100:00080000:3.0:1787828034.754066:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828034.754072:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828034.754086:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828034.754107:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790391040:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828034.754108:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790391168/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828034.754145:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828034.754151:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828034.754153:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828034.754161:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4671 00000100:00080000:3.0:1787828034.754165:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828034.754172:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828034.754178:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828034.754193:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790391296/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828034.754206:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828034.754210:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828034.754213:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828034.754221:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4671 00000100:00080000:3.0:1787828034.754229:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828034.754241:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828034.754246:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790391168:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787828034.754255:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828034.754278:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790391296:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828034.754290:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790391424/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828034.754324:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828034.754332:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828034.754336:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828034.754345:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4671 00000100:00100000:1.0:1787828034.754350:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790391424:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787828034.754353:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828034.754364:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828034.754400:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828034.754423:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790391552/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828034.754449:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828034.754456:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828034.754461:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828034.754470:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4671 00000100:00080000:3.0:1787828034.754480:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828034.754489:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790391552:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787828034.754494:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828034.754508:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828034.754541:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790391680/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828034.754575:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828034.754581:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828034.754585:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828034.754594:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4671 00000100:00100000:1.0:1787828034.754605:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790391680:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828034.754606:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828034.754629:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828034.754645:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828034.754674:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790391808/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828034.754741:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790391808:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828034.754746:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828034.754752:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828034.754755:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828034.754763:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4671 00000100:00080000:3.0:1787828034.754772:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828034.754781:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828034.754790:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828034.754809:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790391936/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828034.754834:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828034.754842:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828034.754846:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828034.754855:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4671 00000100:00080000:3.0:1787828034.754863:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828034.754868:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790391936:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787828034.754881:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828034.754894:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828034.754924:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790392064/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828034.754993:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790392064:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828035.265825:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828035.457993:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828036.289803:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828036.481958:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828037.313822:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828037.505948:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787828037.826086:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:0.0:1787828037.826161:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787828037.826188:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828037.826268:0:2387:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:0.0:1787828037.826283:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828037.826321:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828037.826336:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828037.826391:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828037.826405:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828037.826419:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828037.826434:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:1.0:1787828037.826436:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828034/real 1787828037] req@ffff9e680a2d6d00 x1874668790392064/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828037.826474:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790392064/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828037.826489:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828037.826495:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828037.826501:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790392064:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828037.826510:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828034/real 1787828037] req@ffff9e680a2d4780 x1874668790391936/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828037.826527:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790391936/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828037.826537:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828037.826542:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828037.826547:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790391936:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828037.826556:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828034/real 1787828037] req@ffff9e680a2d7c00 x1874668790391808/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828037.826574:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790391808/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828037.826587:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828037.826592:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828037.826597:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790391808:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828037.826606:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828034/real 1787828037] req@ffff9e680a2d7480 x1874668790391680/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828037.826625:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790391680/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828037.826639:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828037.826645:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828037.826649:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790391680:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828037.826678:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828034/real 1787828037] req@ffff9e680a2d5e00 x1874668790391552/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828037.826731:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790391552/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828037.826745:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828037.826750:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828037.826755:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790391552:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828037.826764:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828034/real 1787828037] req@ffff9e680a2d61c0 x1874668790391424/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828037.826782:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790391424/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828037.826795:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828037.826799:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828037.826804:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790391424:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828037.826813:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828034/real 1787828037] req@ffff9e680a2d6580 x1874668790391168/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828037.826832:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790391168/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828037.826845:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828037.826851:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828037.826857:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790391168:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828037.826874:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828034/real 1787828037] req@ffff9e680a2d4000 x1874668790391296/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828037.826890:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790391296/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828037.826901:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828037.826906:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828037.826911:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790391296:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828037.826920:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828034/real 1787828037] req@ffff9e680a2d5a40 x1874668790391040/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828037.826939:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790391040/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828037.826951:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828037.826956:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828037.826963:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790391040:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828038.337815:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828038.530006:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828039.361807:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828039.553956:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828039.873813:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828039.873826:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828039.873832:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828039.873846:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4676 00000100:00080000:3.0:1787828039.873861:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828039.873871:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828039.873919:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790392192/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828039.873965:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828039.873971:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828039.873974:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828039.873979:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4676 00000100:00080000:3.0:1787828039.873983:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828039.873988:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828039.873999:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828039.874014:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790392320/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828039.874027:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828039.874031:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828039.874033:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828039.874039:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4676 00000100:00080000:3.0:1787828039.874042:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828039.874046:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828039.874052:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828039.874064:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790392448/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828039.874075:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828039.874080:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828039.874082:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828039.874088:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4676 00000100:00080000:3.0:1787828039.874092:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828039.874097:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828039.874103:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828039.874113:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790392192:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828039.874121:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790392576/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828039.874140:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828039.874144:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828039.874146:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828039.874150:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4676 00000100:00080000:3.0:1787828039.874156:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828039.874160:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828039.874166:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828039.874191:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790392704/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828039.874205:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828039.874213:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828039.874217:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828039.874225:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4676 00000800:00000100:1.0:1787828039.874235:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828039.874239:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828039.874255:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828039.874266:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828039.874296:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790392832/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828039.874312:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828039/real 1787828039] req@ffff9e680a2d70c0 x1874668790392192/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828039.874325:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828039.874370:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828039.874375:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828039.874385:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4676 00000100:00080000:3.0:1787828039.874393:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828039.874400:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828039.874426:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828039.874445:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790392960/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828039.874462:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828039.874467:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828039.874470:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828039.874476:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4676 00000100:00080000:3.0:1787828039.874482:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828039.874489:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828039.874497:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828039.874515:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790393088/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828039.874527:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828039.874531:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828039.874534:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828039.874539:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4676 00000100:00080000:3.0:1787828039.874546:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828039.874552:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828039.874559:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828039.874583:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790393216/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828039.874601:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790392192/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828039.874624:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828039.874632:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828039.874641:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790392192:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828039.874655:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790392320:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828039.874746:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828039.874774:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828039/real 1787828039] req@ffff9e680a2d4f00 x1874668790392320/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828039.874797:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790392320/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828039.874813:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828039.874819:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828039.874825:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790392320:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828039.874837:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790392448:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828039.874875:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828039.874897:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828039/real 1787828039] req@ffff9e680a2d52c0 x1874668790392448/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828039.874918:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790392448/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828039.874933:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828039.874939:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828039.874944:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790392448:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828039.875013:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790392576:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828039.875070:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828039.875089:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828039/real 1787828039] req@ffff9e680a2d4b40 x1874668790392576/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828039.875105:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790392576/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828039.875117:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828039.875122:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828039.875127:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790392576:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828039.875138:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790392704:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828039.875165:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828039.875180:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828039/real 1787828039] req@ffff9e680a2d43c0 x1874668790392704/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828039.875197:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790392704/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828039.875211:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828039.875217:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828039.875222:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790392704:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828039.875232:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790392832:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828039.875258:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828039.875275:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828039/real 1787828039] req@ffff9e680a2d5a40 x1874668790392832/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828039.875292:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790392832/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828039.875306:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828039.875311:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828039.875315:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790392832:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828039.875323:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790392960:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828039.875378:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828039.875410:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828039/real 1787828039] req@ffff9e680a2d4000 x1874668790392960/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828039.875426:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790392960/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828039.875444:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828039.875449:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828039.875453:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790392960:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828039.875462:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790393088:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828039.875489:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828039.875505:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828039/real 1787828039] req@ffff9e680a2d6580 x1874668790393088/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828039.875521:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790393088/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828039.875532:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828039.875536:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828039.875541:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790393088:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828039.875549:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790393216:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828039.875577:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828039.875592:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828039/real 1787828039] req@ffff9e680a2d61c0 x1874668790393216/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828039.875609:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790393216/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828039.875620:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828039.875624:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828039.875630:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790393216:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828040.385798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828040.577969:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828041.409812:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828041.601995:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828042.433820:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828042.625936:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828043.457856:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828043.649970:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828044.481829:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828044.674005:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828044.993909:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828044.993925:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828044.993931:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828044.993943:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4681 00000100:00080000:0.0:1787828044.993957:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828044.993965:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828044.994012:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790393344/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828044.994053:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828044.994058:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828044.994068:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828044.994074:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4681 00000100:00080000:0.0:1787828044.994078:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828044.994083:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828044.994095:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828044.994112:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790393472/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828044.994127:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828044.994132:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828044.994135:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828044.994143:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790393344:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828044.994145:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4681 00000100:00080000:0.0:1787828044.994156:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828044.994164:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828044.994170:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828044.994185:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790393600/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828044.994200:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828044.994205:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828044.994207:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828044.994212:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4681 00000100:00080000:0.0:1787828044.994232:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828044.994237:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828044.994243:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828044.994254:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828044.994262:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790393728/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828044.994278:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828044.994284:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828044.994286:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828044.994292:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4681 00000100:00080000:0.0:1787828044.994321:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828044.994326:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828044.994333:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828044.994354:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790393856/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828044.994365:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828044/real 1787828044] req@ffff9e6803e99680 x1874668790393344/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828044.994372:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828044.994384:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828044.994389:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828044.994405:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4681 00000100:00080000:0.0:1787828044.994413:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828044.994419:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828044.994426:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828044.994429:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790393344/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828044.994445:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790393984/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828044.994451:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828044.994469:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787828044.994476:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828044.994483:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790393344:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828044.994486:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828044.994494:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828044.994504:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4681 00000100:00080000:0.0:1787828044.994508:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828044.994513:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828044.994517:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828044.994530:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790393472:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828044.994536:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790394112/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828044.994555:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828044.994560:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828044.994563:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828044.994568:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4681 00000100:00080000:0.0:1787828044.994573:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828044.994581:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828044.994583:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828044.994594:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828044.994613:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828044/real 1787828044] req@ffff9e6803e99e00 x1874668790393472/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828044.994614:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790394240/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828044.994681:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828044.994685:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828044.994689:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828044.994734:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4681 00000100:00080000:0.0:1787828044.994739:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828044.994745:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828044.994752:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828044.994767:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790394368/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828044.994773:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790393472/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828044.994801:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828044.994808:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828044.994816:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790393472:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828044.994839:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790393600:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828044.994874:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828044.994891:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828044/real 1787828044] req@ffff9e6803e98f00 x1874668790393600/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828044.994907:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790393600/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828044.994917:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828044.994921:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828044.994925:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790393600:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828044.994932:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790393728:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828044.994951:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828044.994962:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828044/real 1787828044] req@ffff9e6803e9b840 x1874668790393728/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828044.994975:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790393728/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828044.994985:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828044.994988:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828044.994992:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790393728:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828044.994999:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790393856:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828044.995016:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828044.995026:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828044/real 1787828044] req@ffff9e6803e9a940 x1874668790393856/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828044.995040:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790393856/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828044.995050:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828044.995054:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828044.995058:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790393856:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828044.995065:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790393984:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828044.995082:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828044.995093:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828044/real 1787828044] req@ffff9e6803e9a580 x1874668790393984/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828044.995105:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790393984/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828044.995116:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828044.995120:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828044.995124:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790393984:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828044.995149:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790394112:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828044.995168:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828044.995179:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828044/real 1787828044] req@ffff9e6803e9a1c0 x1874668790394112/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828044.995193:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790394112/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828044.995202:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828044.995205:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828044.995210:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790394112:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828044.995217:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790394240:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828044.995235:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828044.995246:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828044/real 1787828044] req@ffff9e6803e9b480 x1874668790394240/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828044.995259:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790394240/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828044.995269:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828044.995272:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828044.995276:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790394240:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828044.995282:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790394368:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828044.995326:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828044.995343:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828044/real 1787828044] req@ffff9e6803e98780 x1874668790394368/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828044.995357:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790394368/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828044.995366:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828044.995370:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828044.995374:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790394368:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828045.505795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828045.697949:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828046.529807:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828046.721982:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828047.553817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828047.746029:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828048.577827:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828048.769942:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828049.601833:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828049.793967:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828050.113929:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828050.113958:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828050.113963:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828050.113977:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4686 00000100:00080000:3.0:1787828050.113990:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828050.114018:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828050.114083:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790394496/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828050.114161:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828050.114170:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828050.114176:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828050.114190:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4686 00000100:00080000:3.0:1787828050.114203:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828050.114214:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828050.114243:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790394496:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787828050.114251:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828050.114326:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790394624/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828050.114345:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828050.114363:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828050.114414:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828050/real 1787828050] req@ffff9e680a2d5e00 x1874668790394496/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828050.114432:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790394496/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828050.114439:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828050.114447:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828050.114454:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828050.114459:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4686 00000100:00080000:2.0:1787828050.114464:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828050.114489:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790394496:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828050.114490:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828050.114536:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790394624:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828050.114592:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828050.114621:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828050/real 1787828050] req@ffff9e680a2d7480 x1874668790394624/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828050.114644:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790394624/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828050.114659:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828050.114664:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828050.114669:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790394624:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828050.114777:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828050.114863:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828050.114887:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790394752/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828050.114917:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828050.114923:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828050.114926:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828050.114950:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4686 00000100:00080000:3.0:1787828050.114957:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828050.114965:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790394752:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828050.114967:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828050.114978:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828050.114998:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790394880/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828050.115032:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828050.115043:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828050.115052:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828050.115056:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828050.115060:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828050/real 1787828050] req@ffff9e680a2d7c00 x1874668790394752/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828050.115071:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4686 00000100:00080000:3.0:1787828050.115088:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828050.115091:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790394752/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828050.115104:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787828050.115109:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828050.115118:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:3.0:1787828050.115121:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828050.115123:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790394752:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828050.115154:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790394880:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828050.115162:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790395008/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828050.115188:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828050.115189:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828050.115199:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828050.115205:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828050.115206:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828050/real 1787828050] req@ffff9e680a2d4780 x1874668790394880/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828050.115224:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4686 00000100:00100000:2.0:1787828050.115237:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790394880/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828050.115238:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828050.115252:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828050.115303:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828050.115306:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828050.115314:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828050.115319:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790394880:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828050.115326:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790395136/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828050.115340:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790395008:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787828050.115343:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828050.115349:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828050.115352:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828050.115359:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4686 00000100:00080000:3.0:1787828050.115365:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828050.115367:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828050.115375:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828050.115382:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828050.115390:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828050/real 1787828050] req@ffff9e680a2d6d00 x1874668790395008/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828050.115399:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790395264/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828050.115418:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828050.115420:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790395008/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828050.115426:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828050.115430:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828050.115441:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4686 00000100:00080000:2.0:1787828050.115443:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828050.115446:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828050.115452:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828050.115457:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828050.115464:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790395008:192.168.203.109@tcp:8:kworker.0 00000008:00100000:3.0:1787828050.115467:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828050.115485:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790395136:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828050.115493:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790395392/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828050.115508:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828050.115513:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828050.115515:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828050.115521:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828050.115526:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4686 00000100:00080000:3.0:1787828050.115551:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828050.115557:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828050.115565:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828050.115577:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828050/real 1787828050] req@ffff9e680a2d7c00 x1874668790395136/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828050.115584:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790395520/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828050.115636:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790395136/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828050.115652:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828050.115656:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828050.115661:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790395136:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828050.115668:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790395264:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828050.115689:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828050.115701:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828050/real 1787828050] req@ffff9e680a2d7480 x1874668790395264/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828050.115764:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790395264/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828050.115782:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828050.115787:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828050.115792:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790395264:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828050.115817:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790395392:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828050.115847:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828050.115862:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828050/real 1787828050] req@ffff9e680a2d5e00 x1874668790395392/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828050.115876:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790395392/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828050.115885:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828050.115888:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828050.115892:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790395392:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828050.115899:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790395520:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828050.115921:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828050.115935:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828050/real 1787828050] req@ffff9e680a2d61c0 x1874668790395520/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828050.115949:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790395520/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828050.115958:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828050.115962:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828050.115966:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790395520:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828050.625777:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828050.817863:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828051.649809:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828051.841952:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828052.673825:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828052.865938:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828053.697787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828053.889918:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828054.721824:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828054.913996:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828055.233859:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828055.233871:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828055.233877:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828055.233889:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4691 00000100:00080000:0.0:1787828055.233903:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828055.233912:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828055.233960:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790395648/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828055.233999:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828055.234004:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828055.234007:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828055.234012:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4691 00000100:00080000:0.0:1787828055.234016:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828055.234021:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828055.234035:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828055.234051:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790395776/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828055.234068:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828055.234080:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828055.234083:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790395648:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828055.234100:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828055.234109:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4691 00000100:00080000:0.0:1787828055.234113:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828055.234120:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828055.234129:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828055.234146:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790395904/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828055.234160:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828055.234165:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828055.234168:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828055.234174:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4691 00000100:00080000:0.0:1787828055.234178:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828055.234185:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828055.234192:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828055.234205:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790396032/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828055.234219:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828055.234256:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828055.234262:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828055.234266:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828055.234277:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4691 00000100:00080000:0.0:1787828055.234292:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828055.234297:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828055.234304:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828055.234332:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828055/real 1787828055] req@ffff9e6803e992c0 x1874668790395648/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828055.234339:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790396160/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828055.234424:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828055.234428:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828055.234431:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828055.234439:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790395648/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828055.234446:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4691 00000100:00080000:0.0:1787828055.234462:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828055.234484:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787828055.234487:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787828055.234492:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828055.234498:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828055.234513:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790395648:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787828055.234523:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790396288/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828055.234538:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828055.234543:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828055.234546:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828055.234551:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4691 00000100:00080000:0.0:1787828055.234555:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828055.234563:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790395776:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828055.234565:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828055.234589:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828055.234612:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790396416/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828055.234629:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828055.234629:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828055.234642:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828055.234646:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828055.234653:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4691 00000100:00100000:2.0:1787828055.234662:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828055/real 1787828055] req@ffff9e6803e9bc00 x1874668790395776/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828055.234665:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828055.234681:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828055.234731:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828055.234750:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790396544/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828055.234791:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828055.234797:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828055.234800:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828055.234809:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790395776/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828055.234815:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4691 00000100:00080000:0.0:1787828055.234834:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828055.234851:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787828055.234855:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787828055.234859:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828055.234868:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828055.234876:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790395776:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828055.234882:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790396672/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828055.234896:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790395904:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828055.234957:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828055.234975:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828055/real 1787828055] req@ffff9e6803e98b40 x1874668790395904/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828055.234989:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790395904/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828055.234999:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828055.235002:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828055.235006:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790395904:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828055.235013:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790396032:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828055.235034:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828055.235044:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828055/real 1787828055] req@ffff9e6803e9ad00 x1874668790396032/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828055.235056:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790396032/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828055.235064:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828055.235067:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828055.235070:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790396032:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828055.235077:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790396160:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828055.235093:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828055.235104:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828055/real 1787828055] req@ffff9e6803e98780 x1874668790396160/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828055.235116:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790396160/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828055.235125:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828055.235129:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828055.235133:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790396160:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828055.235139:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790396288:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828055.235156:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828055.235165:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828055/real 1787828055] req@ffff9e6803e9b480 x1874668790396288/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828055.235176:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790396288/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828055.235186:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828055.235190:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828055.235193:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790396288:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828055.235217:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790396416:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828055.235267:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828055.235286:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828055/real 1787828055] req@ffff9e6803e9a1c0 x1874668790396416/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828055.235305:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790396416/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828055.235317:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828055.235320:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828055.235325:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790396416:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828055.235332:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790396544:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828055.235358:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828055.235370:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828055/real 1787828055] req@ffff9e6803e9a580 x1874668790396544/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828055.235391:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790396544/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828055.235400:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828055.235404:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828055.235407:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790396544:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828055.235414:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790396672:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828055.235431:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828055.235441:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828055/real 1787828055] req@ffff9e6803e9a940 x1874668790396672/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828055.235453:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790396672/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828055.235461:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828055.235464:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828055.235467:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790396672:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828055.745839:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828055.938066:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828056.769787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828056.962083:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828057.793863:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828057.986908:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828058.817876:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828059.009937:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828059.841819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828060.034133:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828060.353874:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828060.353885:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828060.353891:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828060.353904:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4697 00000100:00080000:3.0:1787828060.353917:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828060.353926:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828060.353972:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790396800/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828060.354021:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828060.354025:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828060.354028:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828060.354058:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4697 00000100:00080000:3.0:1787828060.354061:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828060.354065:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828060.354074:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828060.354089:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790396928/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828060.354108:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828060.354113:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790396800:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828060.354114:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828060.354119:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828060.354151:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4697 00000100:00080000:3.0:1787828060.354156:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828060.354161:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828060.354168:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828060.354185:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790397056/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828060.354246:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828060.354250:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828060.354252:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828060.354257:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4697 00000100:00080000:3.0:1787828060.354262:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828060.354267:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787828060.354275:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787828060.354279:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828060.354297:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790397184/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828060.354319:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828060.354323:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828060.354324:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828060.354329:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4697 00000100:00080000:3.0:1787828060.354334:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828060.354341:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828060/real 1787828060] req@ffff9e680a2d6580 x1874668790396800/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828060.354342:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828060.354353:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828060.354374:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790397312/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828060.354378:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790396800/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828060.354407:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828060.354412:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828060.354412:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828060.354415:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828060.354422:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828060.354425:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4697 00000100:00100000:2.0:1787828060.354433:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790396800:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828060.354433:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828060.354447:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828060.354454:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828060.354464:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790397440/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828060.354475:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790396928:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828060.354476:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828060.354486:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828060.354488:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828060.354494:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4697 00000100:00080000:3.0:1787828060.354498:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828060.354501:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828060.354505:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828060.354522:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790397568/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828060.354539:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828060.354541:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828060.354547:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828060.354550:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828060.354556:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4697 00000100:00080000:3.0:1787828060.354560:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828060.354561:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828060/real 1787828060] req@ffff9e680a2d4000 x1874668790396928/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828060.354568:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828060.354579:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828060.354590:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790396928/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828060.354596:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790397696/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828060.354624:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828060.354638:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828060.354644:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828060.354646:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828060.354652:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828060.354656:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790396928:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828060.354662:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4697 00000100:00080000:3.0:1787828060.354669:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828060.354674:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790397056:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828060.354678:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828060.354688:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828060.354702:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790397824/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828060.354739:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828060.354755:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828060/real 1787828060] req@ffff9e680a2d5a40 x1874668790397056/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828060.354787:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790397056/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828060.354820:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828060.354827:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828060.354864:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790397056:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828060.354879:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790397184:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828060.354933:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828060.354960:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828060/real 1787828060] req@ffff9e680a2d43c0 x1874668790397184/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828060.354973:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790397184/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828060.354981:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828060.354984:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828060.354987:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790397184:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828060.354993:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790397312:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828060.355013:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828060.355044:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828060/real 1787828060] req@ffff9e680a2d4b40 x1874668790397312/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828060.355058:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790397312/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828060.355068:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828060.355071:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828060.355076:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790397312:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828060.355100:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790397440:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828060.355122:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828060.355134:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828060/real 1787828060] req@ffff9e680a2d52c0 x1874668790397440/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828060.355148:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790397440/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828060.355160:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828060.355163:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828060.355167:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790397440:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828060.355173:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790397568:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828060.355246:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828060.355260:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828060/real 1787828060] req@ffff9e680a2d4f00 x1874668790397568/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828060.355272:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790397568/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828060.355298:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828060.355302:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828060.355305:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790397568:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828060.355313:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790397696:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828060.355343:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828060.355356:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828060/real 1787828060] req@ffff9e680a2d70c0 x1874668790397696/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828060.355370:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790397696/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828060.355380:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828060.355383:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828060.355386:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790397696:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828060.355393:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790397824:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828060.355412:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828060.355422:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828060/real 1787828060] req@ffff9e680a2d6580 x1874668790397824/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828060.355436:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790397824/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828060.355456:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828060.355459:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828060.355463:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790397824:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828060.865770:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828061.057930:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828061.889819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828062.081921:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828062.913831:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828063.105970:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828063.937810:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828064.129894:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828064.961894:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828065.153957:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828065.473816:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828065.473841:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828065.473849:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828065.473864:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4702 00000100:00080000:1.0:1787828065.473881:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828065.473893:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828065.473964:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790397952/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828065.474012:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828065.474019:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828065.474022:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828065.474031:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4702 00000100:00080000:1.0:1787828065.474040:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828065.474046:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828065.474061:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828065.474080:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790397952:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828065.474089:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790398080/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828065.474116:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828065.474124:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828065.474128:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828065.474137:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4702 00000100:00080000:1.0:1787828065.474147:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828065.474175:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828065.474186:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828065.474210:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790398208/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828065.474218:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828065.474254:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828065.474259:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828065.474262:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828065.474269:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4702 00000100:00080000:1.0:1787828065.474275:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828065.474287:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828065.474288:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828065/real 1787828065] req@ffff9e6831aa9a40 x1874668790397952/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828065.474297:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828065.474332:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790398336/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828065.474339:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790397952/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828065.474397:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828065.474404:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828065.474407:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828065.474426:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828065.474428:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790397952:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828065.474432:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828065.474447:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4702 00000100:00080000:1.0:1787828065.474455:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828065.474462:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828065.474471:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828065.474473:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790398080:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828065.474508:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790398464/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828065.474520:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828065.474545:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828065/real 1787828065] req@ffff9e6831aaad00 x1874668790398080/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828065.474551:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828065.474574:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828065.474584:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828065.474589:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790398080/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828065.474603:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4702 00000100:00080000:2.0:1787828065.474609:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828065.474617:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828065.474618:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828065.474634:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790398080:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828065.474639:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828065.474650:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790398208:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787828065.474655:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828065.474680:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828065.474739:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828065/real 1787828065] req@ffff9e6831aabc00 x1874668790398208/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828065.474754:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790398592/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828065.474758:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790398208/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828065.474846:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828065.474847:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828065.474865:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828065.474867:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828065.474875:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828065.474884:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790398208:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828065.474893:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4702 00000100:00080000:1.0:1787828065.474909:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828065.474911:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790398336:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787828065.474920:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828065.474933:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828065.474953:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828065.474966:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790398720/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828065.474975:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828065/real 1787828065] req@ffff9e6831aab480 x1874668790398336/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828065.475048:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828065.475048:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790398336/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828065.475074:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828065.475092:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828065.475093:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828065.475109:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828065.475121:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4702 00000100:00100000:2.0:1787828065.475122:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790398336:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787828065.475149:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828065.475186:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828065.475196:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790398464:192.168.203.109@tcp:8:kworker.0 00000008:00100000:1.0:1787828065.475205:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828065.475229:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828065.475241:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790398848/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828065.475251:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828065/real 1787828065] req@ffff9e6831aa9a40 x1874668790398464/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828065.475345:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790398464/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828065.475359:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828065.475362:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828065.475377:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828065.475380:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828065.475385:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790398464:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787828065.475390:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828065.475399:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790398592:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828065.475411:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4702 00000100:00080000:1.0:1787828065.475428:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828065.475435:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787828065.475449:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1787828065.475458:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828065.475468:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828065/real 1787828065] req@ffff9e6831aa8000 x1874668790398592/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828065.475485:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790398976/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828065.475498:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790398592/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828065.475554:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828065.475561:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828065.475569:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790398592:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828065.475582:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790398720:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828065.475613:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828065.475625:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828065/real 1787828065] req@ffff9e6831aaa1c0 x1874668790398720/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828065.475639:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790398720/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828065.475648:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828065.475652:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828065.475655:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790398720:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828065.475674:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790398848:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828065.475722:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828065.475737:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828065/real 1787828065] req@ffff9e6831aa8780 x1874668790398848/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828065.475751:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790398848/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828065.475770:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828065.475774:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828065.475778:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790398848:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828065.475786:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790398976:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828065.475808:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828065.475819:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828065/real 1787828065] req@ffff9e6831aaa940 x1874668790398976/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828065.475832:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790398976/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828065.475842:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828065.475845:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828065.475849:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790398976:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828065.985848:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828066.177916:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828067.009795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828067.201945:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828068.033837:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828068.225929:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828069.057835:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828069.249973:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828070.081850:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828070.273958:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828070.593837:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828070.593848:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828070.593854:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828070.593866:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4707 00000100:00080000:3.0:1787828070.593879:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828070.593888:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828070.593963:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790399104/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828070.593988:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828070.593991:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828070.593993:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828070.593997:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4707 00000100:00080000:3.0:1787828070.594000:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828070.594021:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828070.594033:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828070.594052:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790399232/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828070.594068:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828070.594072:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828070.594075:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828070.594085:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790399104:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828070.594085:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4707 00000100:00080000:3.0:1787828070.594095:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828070.594114:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828070.594155:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828070.594169:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790399360/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828070.594178:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828070.594181:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828070.594183:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828070.594187:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4707 00000100:00080000:3.0:1787828070.594198:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828070.594221:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828070.594228:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828070.594241:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828070.594249:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790399488/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828070.594264:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828070.594270:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828070.594273:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828070.594279:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4707 00000100:00080000:3.0:1787828070.594284:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828070.594289:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828070.594295:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828070.594323:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828070/real 1787828070] req@ffff9e680a2d61c0 x1874668790399104/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828070.594324:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790399616/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828070.594345:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828070.594349:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828070.594352:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828070.594365:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4707 00000100:00080000:3.0:1787828070.594386:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828070.594394:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790399104/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828070.594397:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828070.594411:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828070.594427:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828070.594460:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790399744/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828070.594464:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828070.594474:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790399104:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828070.594484:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828070.594489:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828070.594491:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828070.594495:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4707 00000100:00080000:3.0:1787828070.594498:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828070.594501:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828070.594506:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828070.594509:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790399232:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828070.594520:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790399872/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828070.594531:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828070.594535:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828070.594537:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828070.594543:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828070.594544:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4707 00000100:00080000:3.0:1787828070.594552:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828070.594557:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828070.594567:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828070/real 1787828070] req@ffff9e680a2d5e00 x1874668790399232/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828070.594569:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828070.594612:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790400000/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828070.594615:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790399232/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828070.594655:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828070.594662:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828070.594665:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828070.594666:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828070.594697:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828070.594697:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4707 00000100:00080000:3.0:1787828070.594707:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828070.594712:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790399232:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828070.594715:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828070.594733:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790399360:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787828070.594797:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828070.594818:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790400128/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828070.594890:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828070.594921:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828070/real 1787828070] req@ffff9e680a2d7480 x1874668790399360/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828070.594962:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790399360/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828070.594972:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828070.594975:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828070.594978:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790399360:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828070.594985:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790399488:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828070.595022:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828070.595038:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828070/real 1787828070] req@ffff9e680a2d7c00 x1874668790399488/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828070.595054:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790399488/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828070.595064:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828070.595068:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828070.595072:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790399488:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828070.595079:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790399616:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828070.595099:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828070.595112:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828070/real 1787828070] req@ffff9e680a2d6d00 x1874668790399616/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828070.595175:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790399616/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828070.595187:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828070.595192:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828070.595196:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790399616:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828070.595204:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790399744:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828070.595256:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828070.595273:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828070/real 1787828070] req@ffff9e680a2d4780 x1874668790399744/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828070.595289:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790399744/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828070.595309:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828070.595313:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828070.595317:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790399744:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828070.595346:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790399872:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828070.595369:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828070.595381:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828070/real 1787828070] req@ffff9e680a2d61c0 x1874668790399872/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828070.595398:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790399872/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828070.595414:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828070.595420:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828070.595426:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790399872:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828070.595452:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790400000:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828070.595478:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828070.595494:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828070/real 1787828070] req@ffff9e680a2d6580 x1874668790400000/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828070.595508:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790400000/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828070.595519:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828070.595522:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828070.595526:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790400000:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828070.595535:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790400128:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828070.595562:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828070.595579:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828070/real 1787828070] req@ffff9e680a2d70c0 x1874668790400128/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828070.595596:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790400128/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828070.595607:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828070.595611:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828070.595615:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790400128:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828071.105822:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828071.297851:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828072.129757:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828072.321919:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828073.153761:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828073.345830:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828074.177877:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828074.369934:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828075.201754:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828075.393856:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828075.713843:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828075.713855:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828075.713860:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828075.713872:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4712 00000100:00080000:1.0:1787828075.713885:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828075.713894:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828075.713955:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790400256/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828075.714012:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828075.714020:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828075.714024:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828075.714032:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4712 00000100:00080000:1.0:1787828075.714040:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828075.714048:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828075.714062:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828075.714073:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790400256:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828075.714114:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790400384/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828075.714149:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828075.714154:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828075.714157:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828075.714161:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4712 00000100:00080000:1.0:1787828075.714166:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828075.714171:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828075.714177:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828075.714192:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790400512/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828075.714203:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828075.714213:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1787828075.714213:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828075.714220:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828075.714227:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4712 00000100:00080000:1.0:1787828075.714233:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828075.714238:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828075.714246:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828075.714268:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790400640/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828075.714286:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828075.714293:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828075/real 1787828075] req@ffff9e6831aa9e00 x1874668790400256/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828075.714296:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828075.714305:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828075.714322:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4712 00000100:00080000:1.0:1787828075.714330:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828075.714331:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790400256/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828075.714348:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787828075.714360:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787828075.714362:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828075.714371:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828075.714378:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790400256:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828075.714386:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790400768/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828075.714400:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828075.714404:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828075.714406:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828075.714411:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4712 00000100:00080000:1.0:1787828075.714416:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828075.714417:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790400384:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828075.714427:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828075.714438:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828075.714463:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790400896/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828075.714467:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828075.714489:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828075/real 1787828075] req@ffff9e6831aa8f00 x1874668790400384/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828075.714491:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828075.714509:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828075.714513:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828075.714523:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4712 00000100:00080000:1.0:1787828075.714534:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828075.714542:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828075.714549:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828075.714552:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790400384/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828075.714569:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790401024/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828075.714581:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828075.714584:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828075.714594:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828075.714598:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828075.714601:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828075.714607:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4712 00000100:00080000:1.0:1787828075.714615:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828075.714622:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790400384:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828075.714624:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828075.714632:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828075.714651:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790401152/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828075.714651:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790400512:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828075.714675:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828075.714679:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828075.714681:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828075.714686:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4712 00000100:00080000:1.0:1787828075.714690:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828075.714694:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828075.714698:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828075.714744:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828075.714764:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790401280/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828075.714767:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828075/real 1787828075] req@ffff9e6831aa8b40 x1874668790400512/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828075.714822:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790400512/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828075.714845:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828075.714849:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828075.714854:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790400512:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828075.714862:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790400640:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828075.714883:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828075.714895:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828075/real 1787828075] req@ffff9e6831aa9680 x1874668790400640/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828075.714907:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790400640/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828075.714917:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828075.714920:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828075.714924:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790400640:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828075.714929:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790400768:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828075.714948:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828075.714960:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828075/real 1787828075] req@ffff9e6831aaa580 x1874668790400768/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828075.714973:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790400768/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828075.714993:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828075.714997:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828075.715000:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790400768:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828075.715021:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790400896:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828075.715041:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828075.715055:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828075/real 1787828075] req@ffff9e6831aa9e00 x1874668790400896/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828075.715068:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790400896/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828075.715078:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828075.715081:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828075.715106:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790400896:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828075.715114:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790401024:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828075.715140:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828075.715160:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828075/real 1787828075] req@ffff9e6831aaa940 x1874668790401024/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828075.715181:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790401024/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828075.715195:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828075.715201:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828075.715207:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790401024:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828075.715219:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790401152:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828075.715253:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828075.715273:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828075/real 1787828075] req@ffff9e6831aa8780 x1874668790401152/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828075.715295:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790401152/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828075.715305:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828075.715309:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828075.715312:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790401152:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828075.715319:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790401280:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828075.715343:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828075.715356:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828075/real 1787828075] req@ffff9e6831aaa1c0 x1874668790401280/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828075.715367:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790401280/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828075.715376:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828075.715379:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828075.715382:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790401280:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828076.225782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828076.417953:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828077.249861:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828077.441938:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828078.273827:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828078.466004:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828079.297828:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828079.490025:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828080.321870:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828080.514001:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828080.833873:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828080.833886:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828080.833891:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828080.833906:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4717 00000100:00080000:2.0:1787828080.833940:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828080.833953:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828080.834037:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790401408/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828080.834123:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828080.834130:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828080.834135:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828080.834144:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4717 00000100:00080000:2.0:1787828080.834155:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828080.834165:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828080.834182:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828080.834199:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790401408:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787828080.834228:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790401536/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828080.834250:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828080.834255:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828080.834258:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828080.834264:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4717 00000100:00080000:2.0:1787828080.834271:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828080.834277:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828080.834284:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828080.834300:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828080.834304:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790401664/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828080.834320:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828080.834325:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828080.834329:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828080.834335:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4717 00000100:00080000:2.0:1787828080.834340:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828080.834344:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828080.834350:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828080.834365:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790401792/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828080.834384:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828080/real 1787828080] req@ffff9e68044221c0 x1874668790401408/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828080.834394:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828080.834409:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828080.834416:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828080.834427:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4717 00000100:00080000:2.0:1787828080.834434:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828080.834439:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790401408/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828080.834447:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828080.834458:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828080.834472:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787828080.834482:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790401920/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828080.834484:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828080.834499:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790401408:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828080.834517:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828080.834523:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828080.834527:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828080.834535:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4717 00000100:00080000:2.0:1787828080.834538:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828080.834544:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828080.834551:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828080.834552:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790401536:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828080.834575:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790402048/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828080.834593:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828080.834597:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828080.834600:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828080.834605:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4717 00000100:00080000:2.0:1787828080.834610:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828080.834617:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828080.834622:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828080.834631:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828080.834647:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790402176/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828080.834662:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828080/real 1787828080] req@ffff9e68044230c0 x1874668790401536/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828080.834690:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828080.834742:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828080.834745:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828080.834753:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4717 00000100:00080000:2.0:1787828080.834764:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828080.834770:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828080.834777:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828080.834785:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790401536/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828080.834800:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790402304/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828080.834807:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828080.834824:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787828080.834826:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828080.834837:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790401536:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828080.834838:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828080.834846:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828080.834859:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4717 00000100:00080000:2.0:1787828080.834864:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828080.834870:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790401664:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828080.834874:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828080.834888:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828080.834907:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790402432/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828080.834945:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828080.834967:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828080/real 1787828080] req@ffff9e6804422580 x1874668790401664/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828080.834984:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790401664/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828080.835004:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828080.835010:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828080.835017:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790401664:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828080.835028:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790401792:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828080.835089:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828080.835111:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828080/real 1787828080] req@ffff9e6804420b40 x1874668790401792/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828080.835128:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790401792/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828080.835139:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828080.835154:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828080.835158:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790401792:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828080.835166:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790401920:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828080.835190:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828080.835201:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828080/real 1787828080] req@ffff9e6804423480 x1874668790401920/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828080.835215:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790401920/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828080.835224:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828080.835228:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828080.835233:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790401920:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828080.835262:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790402048:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828080.835299:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828080.835322:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828080/real 1787828080] req@ffff9e68044221c0 x1874668790402048/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828080.835348:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790402048/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828080.835366:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828080.835370:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828080.835385:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790402048:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828080.835393:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790402176:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828080.835421:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828080.835435:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828080/real 1787828080] req@ffff9e6804423840 x1874668790402176/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828080.835449:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790402176/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828080.835459:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828080.835463:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828080.835467:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790402176:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828080.835474:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790402304:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828080.835492:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828080.835503:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828080/real 1787828080] req@ffff9e68044212c0 x1874668790402304/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828080.835517:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790402304/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828080.835526:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828080.835529:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828080.835533:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790402304:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828080.835539:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790402432:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828080.835557:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828080.835567:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828080/real 1787828080] req@ffff9e6804421680 x1874668790402432/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828080.835584:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790402432/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828080.835599:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828080.835605:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828080.835611:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790402432:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828081.345827:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828081.537929:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828082.369774:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828082.561905:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828083.393835:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828083.585860:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828084.417806:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828084.609948:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828085.441758:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828085.633966:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828085.953883:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828085.953895:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828085.953901:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828085.953913:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4722 00000100:00080000:1.0:1787828085.953926:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828085.953936:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828085.953985:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790402560/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828085.954051:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828085.954057:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828085.954060:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828085.954065:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4722 00000100:00080000:1.0:1787828085.954069:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828085.954075:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828085.954088:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828085.954105:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790402688/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828085.954120:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828085.954126:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828085.954132:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828085.954134:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790402560:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828085.954140:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4722 00000100:00080000:1.0:1787828085.954150:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828085.954156:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828085.954163:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828085.954179:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790402816/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828085.954191:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828085.954195:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828085.954198:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828085.954205:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4722 00000100:00080000:1.0:1787828085.954210:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828085.954216:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828085.954224:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828085.954241:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828085.954243:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790402944/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828085.954261:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828085.954266:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828085.954269:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828085.954274:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4722 00000100:00080000:1.0:1787828085.954278:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828085.954282:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828085.954288:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828085.954318:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828085/real 1787828085] req@ffff9e6831aa8000 x1874668790402560/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828085.954319:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790403072/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828085.954373:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828085.954387:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828085.954393:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828085.954407:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4722 00000100:00080000:1.0:1787828085.954416:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828085.954423:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828085.954429:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828085.954432:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790402560/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828085.954449:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790403200/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828085.954454:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828085.954473:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828085.954479:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828085.954488:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790402560:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828085.954489:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828085.954496:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828085.954512:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4722 00000100:00080000:1.0:1787828085.954517:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828085.954523:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828085.954528:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828085.954542:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790403328/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828085.954548:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790402688:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828085.954575:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828085.954580:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828085.954583:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828085.954589:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4722 00000100:00080000:1.0:1787828085.954596:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828085.954601:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828085.954607:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828085.954621:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828085.954627:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790403456/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828085.954643:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828085.954648:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828085.954651:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828085.954656:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4722 00000100:00080000:1.0:1787828085.954666:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828085.954667:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828085/real 1787828085] req@ffff9e6831aa9a40 x1874668790402688/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828085.954674:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828085.954687:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828085.954745:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790402688/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828085.954754:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790403584/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828085.954781:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828085.954792:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828085.954799:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790402688:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828085.954810:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790402816:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828085.954846:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828085.954862:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828085/real 1787828085] req@ffff9e6831aab480 x1874668790402816/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828085.954877:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790402816/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828085.954901:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828085.954906:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828085.954914:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790402816:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828085.954925:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790402944:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828085.954957:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828085.954971:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828085/real 1787828085] req@ffff9e6831aabc00 x1874668790402944/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828085.954986:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790402944/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828085.954997:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828085.955000:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828085.955004:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790402944:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828085.955038:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790403072:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828085.955100:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828085.955133:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828085/real 1787828085] req@ffff9e6831aaad00 x1874668790403072/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828085.955173:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790403072/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828085.955186:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828085.955192:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828085.955198:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790403072:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828085.955209:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790403200:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828085.955247:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828085.955274:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828085/real 1787828085] req@ffff9e6831aaa1c0 x1874668790403200/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828085.955297:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790403200/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828085.955321:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828085.955328:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828085.955334:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790403200:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828085.955393:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790403328:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828085.955425:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828085.955440:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828085/real 1787828085] req@ffff9e6831aa8780 x1874668790403328/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828085.955463:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790403328/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828085.955475:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828085.955482:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828085.955486:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790403328:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828085.955501:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790403456:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828085.955524:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828085.955539:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828085/real 1787828085] req@ffff9e6831aaa940 x1874668790403456/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828085.955555:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790403456/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828085.955572:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828085.955579:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828085.955587:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790403456:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828085.955600:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790403584:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828085.955652:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828085.955685:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828085/real 1787828085] req@ffff9e6831aa9e00 x1874668790403584/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828085.955745:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790403584/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828085.955761:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828085.955767:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828085.955773:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790403584:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828086.465805:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828086.657941:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828087.489825:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828087.681997:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828088.513783:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828088.706022:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828089.537855:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828089.730036:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828090.561809:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828090.753920:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828091.073826:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828091.073838:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828091.073844:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828091.073856:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4727 00000100:00080000:1.0:1787828091.073869:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828091.073878:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828091.073928:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790403712/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828091.073965:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828091.073970:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828091.073972:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828091.074005:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4727 00000100:00080000:1.0:1787828091.074012:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828091.074030:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828091.074044:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828091.074053:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790403712:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828091.074067:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790403840/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828091.074084:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828091.074092:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828091.074095:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828091.074100:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4727 00000100:00080000:1.0:1787828091.074104:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828091.074109:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828091.074114:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828091.074128:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790403968/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828091.074141:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828091.074145:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828091.074147:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828091.074155:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4727 00000800:00000100:2.0:1787828091.074159:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828091.074162:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828091.074170:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828091.074176:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828091.074191:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790404096/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828091.074203:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828091.074207:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828091.074209:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828091.074214:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4727 00000100:00080000:1.0:1787828091.074220:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828091.074225:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828091.074232:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828091.074235:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828091/real 1787828091] req@ffff9e6831aaa580 x1874668790403712/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828091.074249:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790404224/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828091.074307:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828091.074315:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787828091.074318:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790403712/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828091.074322:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828091.074342:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4727 00000100:00080000:2.0:1787828091.074347:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828091.074354:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828091.074358:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828091.074375:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790403712:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828091.074383:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828091.074399:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828091.074424:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790404352/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828091.074437:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790403840:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828091.074468:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828091.074476:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828091.074480:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828091.074490:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4727 00000100:00080000:1.0:1787828091.074498:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828091.074508:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787828091.074512:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1787828091.074518:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828091.074540:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790404480/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828091.074555:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828091/real 1787828091] req@ffff9e6831aa9680 x1874668790403840/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828091.074556:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828091.074568:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828091.074575:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828091.074587:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4727 00000100:00080000:1.0:1787828091.074597:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828091.074604:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828091.074610:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828091.074613:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790403840/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828091.074627:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790404608/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828091.074633:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828091.074659:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828091.074660:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828091.074671:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828091.074672:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790403840:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828091.074675:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828091.074690:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4727 00000100:00080000:1.0:1787828091.074732:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828091.074751:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790403968:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828091.074752:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828091.074766:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828091.074789:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790404736/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828091.074822:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828091.074854:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828091/real 1787828091] req@ffff9e6831aa8b40 x1874668790403968/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828091.074881:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790403968/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828091.074894:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828091.074899:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828091.074904:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790403968:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828091.074914:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790404096:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828091.074944:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828091.074959:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828091/real 1787828091] req@ffff9e6831aa8f00 x1874668790404096/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828091.074974:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790404096/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828091.075094:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828091.075101:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828091.075109:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790404096:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828091.075122:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790404224:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828091.075167:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828091.075191:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828091/real 1787828091] req@ffff9e6831aa9e00 x1874668790404224/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828091.075208:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790404224/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828091.075219:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828091.075223:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828091.075228:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790404224:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828091.075252:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790404352:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828091.075279:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828091.075293:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828091/real 1787828091] req@ffff9e6831aaa940 x1874668790404352/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828091.075319:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790404352/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828091.075331:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828091.075335:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828091.075339:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790404352:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828091.075346:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790404480:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828091.075366:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828091.075385:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828091/real 1787828091] req@ffff9e6831aa8780 x1874668790404480/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828091.075408:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790404480/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828091.075424:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828091.075430:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828091.075446:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790404480:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828091.075459:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790404608:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828091.075497:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828091.075519:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828091/real 1787828091] req@ffff9e6831aaa1c0 x1874668790404608/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828091.075543:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790404608/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828091.075575:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828091.075581:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828091.075588:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790404608:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828091.075596:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790404736:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828091.075625:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828091.075650:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828091/real 1787828091] req@ffff9e6831aaad00 x1874668790404736/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828091.075664:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790404736/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828091.075674:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828091.075678:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828091.075682:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790404736:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828091.585842:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828091.777967:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828092.609846:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828092.802054:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828093.633822:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828093.826043:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828094.657811:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828094.849925:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828095.681808:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828095.874043:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828096.193831:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828096.193844:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828096.193850:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828096.193862:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4732 00000100:00080000:1.0:1787828096.193888:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828096.193897:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828096.193977:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790404864/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828096.194033:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828096.194038:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828096.194040:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828096.194046:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4732 00000100:00080000:1.0:1787828096.194051:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828096.194066:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828096.194080:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828096.194102:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790404992/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828096.194130:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828096.194138:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828096.194141:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790404864:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828096.194142:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828096.194151:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4732 00000100:00080000:1.0:1787828096.194160:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828096.194166:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828096.194172:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828096.194189:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790405120/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828096.194202:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828096.194206:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828096.194209:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828096.194215:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4732 00000100:00080000:1.0:1787828096.194220:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828096.194248:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828096.194256:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828096.194259:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828096.194277:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790405248/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828096.194298:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828096.194303:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828096.194305:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828096.194310:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4732 00000100:00080000:1.0:1787828096.194315:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828096.194320:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828096.194327:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828096.194333:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828096/real 1787828096] req@ffff9e6831aabc00 x1874668790404864/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828096.194348:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790405376/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828096.194360:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828096.194375:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828096.194378:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828096.194385:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4732 00000100:00080000:1.0:1787828096.194389:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828096.194392:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790404864/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828096.194400:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828096.194410:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828096.194428:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828096.194433:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790405504/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828096.194440:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828096.194456:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790404864:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828096.194481:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828096.194486:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828096.194489:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828096.194496:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4732 00000100:00080000:1.0:1787828096.194500:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828096.194503:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790404992:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828096.194510:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828096.194521:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828096.194540:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790405632/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828096.194555:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828096.194556:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828096.194565:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828096.194569:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828096.194576:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4732 00000100:00080000:1.0:1787828096.194583:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828096.194584:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828096/real 1787828096] req@ffff9e6831aab480 x1874668790404992/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828096.194599:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828096.194616:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828096.194617:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790404992/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828096.194639:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828096.194644:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828096.194655:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790404992:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828096.194656:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790405760/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828096.194667:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790405120:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828096.194742:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828096.194748:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828096.194752:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828096.194760:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4732 00000800:00000100:2.0:1787828096.194768:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828096.194774:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828096.194786:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828096.194792:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828096/real 1787828096] req@ffff9e6831aa9a40 x1874668790405120/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828096.194798:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828096.194820:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790405120/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828096.194832:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790405888/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828096.194842:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828096.194858:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828096.194867:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790405120:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828096.194877:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790405248:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828096.194906:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828096.194918:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828096/real 1787828096] req@ffff9e6831aa8000 x1874668790405248/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828096.194932:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790405248/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828096.194973:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828096.194979:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828096.194983:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790405248:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828096.194991:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790405376:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828096.195033:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828096.195055:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828096/real 1787828096] req@ffff9e6831aaad00 x1874668790405376/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828096.195078:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790405376/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828096.195095:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828096.195102:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828096.195108:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790405376:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828096.195130:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790405504:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828096.195165:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828096.195181:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828096/real 1787828096] req@ffff9e6831aaa1c0 x1874668790405504/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828096.195196:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790405504/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828096.195220:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828096.195224:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828096.195228:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790405504:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828096.195253:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790405632:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828096.195275:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828096.195286:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828096/real 1787828096] req@ffff9e6831aa8780 x1874668790405632/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828096.195300:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790405632/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828096.195309:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828096.195313:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828096.195316:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790405632:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828096.195323:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790405760:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828096.195351:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828096.195361:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828096/real 1787828096] req@ffff9e6831aaa940 x1874668790405760/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828096.195375:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790405760/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828096.195384:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828096.195387:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828096.195392:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790405760:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828096.195399:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790405888:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828096.195417:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828096.195428:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828096/real 1787828096] req@ffff9e6831aa9e00 x1874668790405888/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828096.195441:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790405888/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828096.195451:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828096.195454:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828096.195458:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790405888:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828096.705805:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828096.897979:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828097.729834:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828097.921979:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828098.753819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828098.945918:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828099.777808:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828099.969981:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828100.801844:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828100.994007:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828101.313903:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828101.313939:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828101.313944:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828101.313957:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4738 00000100:00080000:3.0:1787828101.313971:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828101.313981:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828101.314030:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790406016/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828101.314081:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828101.314089:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828101.314092:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828101.314101:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4738 00000100:00080000:3.0:1787828101.314108:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828101.314117:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828101.314149:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828101.314160:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790406016:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828101.314198:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790406144/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828101.314227:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828101.314235:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828101.314240:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828101.314248:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4738 00000100:00080000:3.0:1787828101.314258:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828101.314267:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828101.314279:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828101.314306:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790406272/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828101.314329:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790406144:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828101.314334:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828101.314345:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828101.314352:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828101.314364:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4738 00000100:00080000:3.0:1787828101.314374:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828101.314385:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828101.314395:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790406272:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787828101.314399:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828101.314437:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790406400/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828101.314479:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828101.314487:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828101.314492:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828101.314503:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4738 00000100:00100000:2.0:1787828101.314512:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790406400:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787828101.314513:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828101.314534:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828101.314548:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828101.314580:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790406528/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828101.314629:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828101.314637:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828101.314641:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828101.314649:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4738 00000100:00080000:3.0:1787828101.314658:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828101.314668:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828101.314679:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828101.314682:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790406528:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828101.314778:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790406656/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828101.314807:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828101.314815:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828101.314821:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790406656:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828101.314823:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828101.314842:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4738 00000100:00080000:3.0:1787828101.314854:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828101.314862:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828101.314885:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828101.314936:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790406784/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828101.314969:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828101.314977:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828101.314981:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828101.314992:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4738 00000100:00080000:3.0:1787828101.315003:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828101.315016:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790406784:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828101.315018:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828101.315039:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828101.315074:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790406912/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828101.315110:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828101.315118:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828101.315122:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828101.315132:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4738 00000100:00080000:3.0:1787828101.315143:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828101.315152:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828101.315160:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790406912:192.168.203.109@tcp:8:kworker.0 00000008:00100000:3.0:1787828101.315168:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828101.315202:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790407040/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828101.315246:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790407040:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828101.825826:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828102.017982:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828102.849830:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828103.042012:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828103.873825:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828104.065969:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787828104.386101:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:0.0:1787828104.386177:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787828104.386204:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828104.386273:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:0.0:1787828104.386291:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828104.386301:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828104.386311:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828104.386324:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828104.386337:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828104.386347:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828104.386359:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787828104.386380:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828101/real 1787828104] req@ffff9e680a2d61c0 x1874668790407040/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828104.386443:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790407040/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828104.386464:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828104.386473:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828104.386481:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790407040:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828104.386495:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828101/real 1787828104] req@ffff9e680a2d6580 x1874668790406912/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828104.386520:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790406912/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828104.386536:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828104.386544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828104.386551:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790406912:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828104.386565:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828101/real 1787828104] req@ffff9e680a2d70c0 x1874668790406784/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828104.386591:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790406784/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828104.386606:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828104.386613:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828104.386619:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790406784:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828104.386632:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828101/real 1787828104] req@ffff9e680a2d4000 x1874668790406656/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828104.386654:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790406656/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828104.386671:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828104.386678:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828104.386685:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790406656:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828104.386752:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828101/real 1787828104] req@ffff9e680a2d5a40 x1874668790406528/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828104.386804:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790406528/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828104.386816:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828104.386821:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828104.386825:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790406528:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828104.386836:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828101/real 1787828104] req@ffff9e680a2d43c0 x1874668790406400/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828104.386859:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790406400/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828104.386875:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828104.386881:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828104.386909:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790406400:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828104.386924:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828101/real 1787828104] req@ffff9e680a2d4b40 x1874668790406272/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828104.386950:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790406272/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828104.386967:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828104.386973:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828104.386980:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790406272:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828104.386992:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828101/real 1787828104] req@ffff9e680a2d52c0 x1874668790406144/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828104.387014:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790406144/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828104.387029:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828104.387035:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828104.387041:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790406144:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828104.387051:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828101/real 1787828104] req@ffff9e680a2d4f00 x1874668790406016/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828104.387071:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790406016/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828104.387085:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828104.387090:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828104.387096:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790406016:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828104.897834:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828105.089898:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828105.921840:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828106.113986:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828106.433819:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828106.433833:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828106.433838:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828106.433852:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4743 00000100:00080000:3.0:1787828106.433865:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828106.433898:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828106.433947:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790407168/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828106.433986:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828106.433991:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828106.433995:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828106.434000:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4743 00000100:00080000:3.0:1787828106.434005:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828106.434010:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828106.434022:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828106.434036:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790407296/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828106.434050:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828106.434055:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828106.434057:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828106.434062:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4743 00000100:00080000:3.0:1787828106.434065:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828106.434070:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828106.434078:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828106.434093:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790407168:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828106.434101:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790407424/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828106.434126:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828106.434133:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828106.434137:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828106.434144:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4743 00000100:00080000:3.0:1787828106.434153:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828106.434162:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828106.434172:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828106.434199:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790407552/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828106.434210:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828106.434232:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828106.434240:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828106.434244:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828106.434253:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4743 00000100:00080000:3.0:1787828106.434262:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828106.434272:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828106.434284:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828106.434287:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828106/real 1787828106] req@ffff9e680a2d4780 x1874668790407168/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828106.434312:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790407680/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828106.434337:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828106.434348:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787828106.434350:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790407168/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828106.434355:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828106.434374:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4743 00000100:00080000:3.0:1787828106.434389:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828106.434395:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828106.434404:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787828106.434414:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:3.0:1787828106.434422:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828106.434428:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790407168:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787828106.434453:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790407296:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828106.434456:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790407808/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828106.434493:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828106.434500:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828106.434504:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828106.434516:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828106.434522:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4743 00000100:00080000:3.0:1787828106.434535:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828106.434543:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828106.434553:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828106.434559:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828106/real 1787828106] req@ffff9e680a2d6d00 x1874668790407296/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828106.434581:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790407936/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828106.434600:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790407296/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828106.434611:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828106.434628:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828106.434637:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828106.434640:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828106.434650:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4743 00000100:00080000:2.0:1787828106.434658:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828106.434664:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828106.434676:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828106.434690:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790407296:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787828106.434732:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828106.434763:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790408064/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828106.434786:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828106.434792:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828106.434797:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828106.434806:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4743 00000100:00080000:3.0:1787828106.434813:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828106.434820:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828106.434831:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828106.434833:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790407424:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828106.434858:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790408192/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828106.434929:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828106.434963:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828106/real 1787828106] req@ffff9e680a2d7c00 x1874668790407424/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828106.434993:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790407424/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828106.435011:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828106.435018:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828106.435024:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790407424:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828106.435038:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790407552:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828106.435080:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828106.435106:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828106/real 1787828106] req@ffff9e680a2d7480 x1874668790407552/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828106.435130:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790407552/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828106.435145:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828106.435151:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828106.435157:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790407552:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828106.435168:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790407680:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828106.435204:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828106.435223:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828106/real 1787828106] req@ffff9e680a2d5e00 x1874668790407680/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828106.435246:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790407680/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828106.435272:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828106.435278:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828106.435284:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790407680:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828106.435295:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790407808:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828106.435333:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828106.435355:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828106/real 1787828106] req@ffff9e680a2d4f00 x1874668790407808/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828106.435381:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790407808/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828106.435402:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828106.435409:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828106.435417:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790407808:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828106.435429:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790407936:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828106.435466:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828106.435488:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828106/real 1787828106] req@ffff9e680a2d52c0 x1874668790407936/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828106.435512:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790407936/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828106.435529:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828106.435535:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828106.435542:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790407936:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828106.435555:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790408064:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828106.435594:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828106.435616:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828106/real 1787828106] req@ffff9e680a2d4b40 x1874668790408064/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828106.435641:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790408064/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828106.435657:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828106.435663:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828106.435669:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790408064:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828106.435754:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790408192:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828106.435813:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828106.435838:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828106/real 1787828106] req@ffff9e680a2d43c0 x1874668790408192/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828106.435861:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790408192/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828106.435899:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828106.435906:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828106.435913:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790408192:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828106.945821:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828107.137979:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828107.969848:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828108.162034:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828108.993847:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828109.186010:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828110.017838:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828110.210055:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828111.041875:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828111.234064:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828111.553957:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828111.553978:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828111.553987:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828111.554003:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4748 00000100:00080000:1.0:1787828111.554037:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828111.554047:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828111.554127:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790408320/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828111.554174:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828111.554180:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828111.554183:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828111.554188:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4748 00000100:00080000:1.0:1787828111.554193:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828111.554198:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828111.554214:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828111.554232:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790408448/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828111.554247:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828111.554260:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828111.554263:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790408320:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828111.554264:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828111.554276:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4748 00000100:00080000:1.0:1787828111.554286:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828111.554291:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828111.554297:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828111.554311:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790408576/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828111.554324:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828111.554328:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828111.554330:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828111.554336:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4748 00000100:00080000:1.0:1787828111.554340:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828111.554344:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828111.554350:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828111.554363:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790408704/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828111.554388:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828111.554391:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828111.554393:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828111.554397:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4748 00000100:00080000:1.0:1787828111.554409:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828111.554413:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828111.554418:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828111.554430:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790408832/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828111.554440:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828111.554446:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828111.554450:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828111.554452:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828111.554476:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4748 00000100:00080000:1.0:1787828111.554482:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828111.554490:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828111.554497:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828111.554510:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790408960/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828111.554521:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828111.554526:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828111.554528:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828111.554533:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4748 00000100:00080000:1.0:1787828111.554538:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828111.554543:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828111.554550:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828111.554559:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828111/real 1787828111] req@ffff9e6831aa8f00 x1874668790408320/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828111.554569:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790409088/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828111.554630:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828111.554634:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828111.554637:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828111.554641:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4748 00000100:00080000:1.0:1787828111.554644:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828111.554652:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828111.554652:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790408320/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828111.554660:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828111.554675:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828111.554676:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790409216/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828111.554685:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828111.554695:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828111.554697:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790408320:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828111.554700:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828111.554704:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828111.554711:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4748 00000100:00080000:1.0:1787828111.554715:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828111.554718:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828111.554722:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828111.554799:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790409344/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828111.554825:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790408448:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828111.554913:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828111.554935:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828111/real 1787828111] req@ffff9e6831aa8b40 x1874668790408448/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828111.554953:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790408448/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828111.554965:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828111.554970:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828111.554974:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790408448:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828111.554983:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790408576:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828111.555007:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828111.555035:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828111/real 1787828111] req@ffff9e6831aa9680 x1874668790408576/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828111.555046:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790408576/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828111.555060:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828111.555063:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828111.555066:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790408576:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828111.555070:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790408704:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828111.555106:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828111.555118:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828111/real 1787828111] req@ffff9e6831aaa580 x1874668790408704/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828111.555132:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790408704/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828111.555143:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828111.555147:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828111.555150:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790408704:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828111.555157:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790408832:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828111.555176:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828111.555186:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828111/real 1787828111] req@ffff9e6831aa9e00 x1874668790408832/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828111.555199:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790408832/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828111.555209:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828111.555214:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828111.555219:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790408832:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828111.555226:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790408960:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828111.555252:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828111.555273:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828111/real 1787828111] req@ffff9e6831aaa940 x1874668790408960/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828111.555296:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790408960/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828111.555316:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828111.555323:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828111.555330:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790408960:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828111.555342:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790409088:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828111.555394:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828111.555410:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828111/real 1787828111] req@ffff9e6831aa8780 x1874668790409088/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828111.555428:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790409088/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828111.555460:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828111.555465:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828111.555471:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790409088:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828111.555484:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790409216:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828111.555525:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828111.555549:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828111/real 1787828111] req@ffff9e6831aaa1c0 x1874668790409216/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828111.555574:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790409216/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828111.555606:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828111.555610:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828111.555615:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790409216:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828111.555645:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790409344:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828111.555670:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828111.555680:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828111/real 1787828111] req@ffff9e6831aaad00 x1874668790409344/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828111.555691:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790409344/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828111.555698:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828111.555701:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828111.555704:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790409344:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828112.065858:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828112.258034:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828113.089853:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828113.282069:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828114.113765:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828114.306074:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828115.137843:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828115.330053:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828116.161783:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828116.353996:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828116.673985:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828116.673996:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828116.674001:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828116.674012:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4753 00000100:00080000:2.0:1787828116.674026:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828116.674034:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828116.674084:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790409472/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828116.674117:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828116.674120:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828116.674122:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828116.674125:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4753 00000100:00080000:2.0:1787828116.674128:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828116.674131:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828116.674141:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828116.674152:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790409600/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828116.674164:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828116.674167:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828116.674169:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828116.674176:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4753 00000100:00100000:1.0:1787828116.674180:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790409472:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828116.674180:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828116.674189:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828116.674206:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828116.674227:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790409728/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828116.674257:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828116.674261:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828116.674264:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828116.674269:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4753 00000100:00080000:2.0:1787828116.674275:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828116.674281:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1787828116.674291:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:2.0:1787828116.674297:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828116.674348:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790409856/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828116.674358:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828116.674362:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828116.674364:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828116.674373:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4753 00000100:00080000:2.0:1787828116.674382:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828116.674386:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828116/real 1787828116] req@ffff9e6804420780 x1874668790409472/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828116.674393:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828116.674411:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828116.674449:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790409472/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828116.674458:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790409984/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828116.674548:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828116.674552:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828116.674561:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787828116.674564:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828116.674573:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828116.674574:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790409472:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828116.674593:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4753 00000100:00080000:2.0:1787828116.674601:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828116.674610:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828116.674620:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828116.674630:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790409600:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828116.674641:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790410112/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828116.674661:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828116.674666:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828116.674678:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828116.674679:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828116.674771:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4753 00000100:00080000:2.0:1787828116.674778:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828116.674784:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828116.674794:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828116/real 1787828116] req@ffff9e6804420f00 x1874668790409600/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828116.674817:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828116.674878:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790410240/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828116.674887:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790409600/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828116.674908:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828116.674925:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828116.674928:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828116.674945:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4753 00000100:00080000:1.0:1787828116.674946:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828116.674955:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828116.674957:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787828116.674964:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828116.674966:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790409600:192.168.203.109@tcp:38:kworker.0 00000008:00100000:2.0:1787828116.674973:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828116.674986:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790409728:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828116.674987:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790410368/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828116.675014:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828116.675018:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828116.675020:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828116.675024:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4753 00000100:00080000:2.0:1787828116.675029:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828116.675035:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828116.675039:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828116.675057:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790410496/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828116.675060:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828116.675103:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828116/real 1787828116] req@ffff9e6804422d00 x1874668790409728/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828116.675127:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790409728/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828116.675149:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828116.675153:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828116.675156:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790409728:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828116.675164:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790409856:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828116.675191:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828116.675202:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828116/real 1787828116] req@ffff9e6804421e00 x1874668790409856/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828116.675213:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790409856/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828116.675221:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828116.675224:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828116.675227:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790409856:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828116.675233:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790409984:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828116.675247:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828116.675256:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828116/real 1787828116] req@ffff9e6804422940 x1874668790409984/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828116.675266:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790409984/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828116.675274:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828116.675277:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828116.675279:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790409984:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828116.675308:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790410112:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828116.675338:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828116.675356:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828116/real 1787828116] req@ffff9e6804420780 x1874668790410112/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828116.675381:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790410112/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828116.675395:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828116.675401:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828116.675406:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790410112:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828116.675418:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790410240:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828116.675449:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828116.675467:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828116/real 1787828116] req@ffff9e6804421680 x1874668790410240/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828116.675488:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790410240/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828116.675502:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828116.675507:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828116.675513:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790410240:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828116.675524:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790410368:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828116.675558:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828116.675591:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828116/real 1787828116] req@ffff9e68044212c0 x1874668790410368/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828116.675605:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790410368/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828116.675622:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828116.675626:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828116.675631:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790410368:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828116.675640:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790410496:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828116.675668:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828116.675685:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828116/real 1787828116] req@ffff9e6804423840 x1874668790410496/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828116.675700:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790410496/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828116.675742:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828116.675749:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828116.675754:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790410496:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828117.185833:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828117.378060:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828118.209806:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828118.402008:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828119.233839:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828119.426060:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828120.257884:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828120.449994:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828121.281787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828121.474041:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828121.793912:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828121.793933:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828121.793944:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828121.793966:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4758 00000100:00080000:2.0:1787828121.793987:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828121.793999:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828121.794079:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790410624/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828121.794122:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828121.794128:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828121.794133:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828121.794141:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4758 00000100:00080000:2.0:1787828121.794150:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828121.794159:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828121.794177:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828121.794188:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790410624:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787828121.794201:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790410752/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828121.794218:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828121.794227:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828121.794229:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828121.794236:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4758 00000100:00080000:2.0:1787828121.794253:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828121.794258:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828121.794265:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828121.794279:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790410880/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828121.794291:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787828121.794314:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828121.794360:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828121.794364:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828121.794373:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828121/real 1787828121] req@ffff9e68044221c0 x1874668790410624/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828121.794381:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4758 00000100:00080000:2.0:1787828121.794398:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828121.794414:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828121.794424:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828121.794427:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790410624/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828121.794455:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787828121.794456:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790411008/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828121.794478:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787828121.794503:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828121.794506:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790410624:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828121.794510:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828121.794530:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828121.794541:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4758 00000100:00080000:2.0:1787828121.794566:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828121.794578:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828121.794586:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790410752:192.168.203.109@tcp:38:kworker.0 00000008:00100000:2.0:1787828121.794592:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828121.794651:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790411136/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828121.794672:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828121.794712:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828121/real 1787828121] req@ffff9e6804423480 x1874668790410752/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828121.794761:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828121.794812:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828121.794818:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828121.794826:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790410752/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828121.794835:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4758 00000100:00080000:2.0:1787828121.794856:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828121.794863:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828121.794883:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787828121.794894:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828121.794905:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790410752:192.168.203.109@tcp:38:kworker.0 00000008:00100000:2.0:1787828121.794908:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828121.794921:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790410880:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828121.794952:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790411264/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828121.794970:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787828121.794982:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828121.794990:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828121.794995:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828121.795008:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4758 00000100:00080000:2.0:1787828121.795017:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828121.795020:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828121/real 1787828121] req@ffff9e6804420b40 x1874668790410880/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828121.795030:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828121.795043:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828121.795067:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790411392/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828121.795093:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828121.795099:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790410880/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828121.795100:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828121.795108:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828121.795123:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4758 00000100:00080000:2.0:1787828121.795134:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828121.795148:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787828121.795149:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787828121.795159:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828121.795166:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828121.795182:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790410880:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828121.795182:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790411520/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828121.795216:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828121.795226:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787828121.795243:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790411008:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787828121.795243:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828121.795263:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4758 00000100:00080000:2.0:1787828121.795270:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828121.795278:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828121.795288:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828121.795306:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790411648/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828121.795314:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828121.795350:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828121/real 1787828121] req@ffff9e6804422580 x1874668790411008/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828121.795370:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790411008/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828121.795383:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828121.795389:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828121.795395:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790411008:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828121.795424:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790411136:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828121.795446:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828121.795458:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828121/real 1787828121] req@ffff9e68044230c0 x1874668790411136/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828121.795473:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790411136/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828121.795484:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828121.795488:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828121.795492:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790411136:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828121.795499:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790411264:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828121.795553:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828121.795565:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828121/real 1787828121] req@ffff9e68044221c0 x1874668790411264/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828121.795579:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790411264/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828121.795599:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828121.795602:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828121.795605:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790411264:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828121.795631:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790411392:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828121.795660:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828121.795675:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828121/real 1787828121] req@ffff9e6804423840 x1874668790411392/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828121.795689:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790411392/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828121.795699:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828121.795703:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828121.795707:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790411392:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828121.795714:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790411520:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828121.795736:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828121.795753:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828121/real 1787828121] req@ffff9e68044212c0 x1874668790411520/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828121.795838:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790411520/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828121.795851:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828121.795856:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828121.795862:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790411520:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828121.795873:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790411648:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828121.795903:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828121.795917:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828121/real 1787828121] req@ffff9e6804421680 x1874668790411648/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828121.795932:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790411648/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828121.795942:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828121.795946:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828121.795949:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790411648:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828122.305863:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828122.497999:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828123.329855:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828123.521986:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828124.353858:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828124.546176:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828125.377942:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828125.569913:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828126.401787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828126.594003:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828126.913918:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828126.913929:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828126.913934:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828126.913945:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4763 00000100:00080000:0.0:1787828126.913958:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828126.913968:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828126.914026:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790411776/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828126.914053:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828126.914056:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828126.914058:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828126.914062:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4763 00000100:00080000:0.0:1787828126.914065:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828126.914069:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828126.914078:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828126.914089:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790411904/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828126.914104:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828126.914107:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828126.914109:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828126.914114:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4763 00000100:00080000:0.0:1787828126.914117:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828126.914120:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828126.914123:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828126.914131:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790412032/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828126.914142:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790411776:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828126.914166:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828126.914171:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828126.914174:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828126.914182:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4763 00000100:00080000:0.0:1787828126.914205:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828126.914211:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828126.914217:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828126.914219:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828126.914234:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790412160/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828126.914245:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828126.914248:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828126.914250:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828126.914253:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4763 00000100:00080000:0.0:1787828126.914258:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828126.914262:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828126.914267:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828126.914277:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828126/real 1787828126] req@ffff9e6803e9b840 x1874668790411776/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828126.914282:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790412288/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828126.914312:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828126.914315:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828126.914319:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828126.914321:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790411776/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828126.914328:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4763 00000100:00080000:0.0:1787828126.914335:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828126.914347:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787828126.914347:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787828126.914358:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787828126.914367:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828126.914374:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790411776:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787828126.914379:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790412416/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828126.914387:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790411904:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828126.914407:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828126.914412:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828126.914416:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828126.914423:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4763 00000800:00000100:1.0:1787828126.914425:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828126.914433:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828126.914443:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828126.914445:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828126/real 1787828126] req@ffff9e6803e98f00 x1874668790411904/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828126.914456:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828126.914462:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790411904/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828126.914474:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790412544/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828126.914478:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828126.914490:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787828126.914502:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828126.914503:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790411904:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828126.914510:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828126.914515:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828126.914521:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4763 00000100:00080000:0.0:1787828126.914524:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828126.914528:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828126.914531:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828126.914541:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790412672/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828126.914549:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828126.914552:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828126.914554:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828126.914559:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790412032:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828126.914560:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4763 00000100:00080000:0.0:1787828126.914567:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828126.914571:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828126.914576:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828126.914589:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790412800/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828126.914599:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828126.914617:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828126/real 1787828126] req@ffff9e6803e99e00 x1874668790412032/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828126.914629:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790412032/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828126.914636:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828126.914640:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828126.914642:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790412032:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828126.914648:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790412160:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828126.914662:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828126.914670:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828126/real 1787828126] req@ffff9e6803e99680 x1874668790412160/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828126.914679:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790412160/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828126.914686:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828126.914689:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828126.914691:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790412160:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828126.914745:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790412288:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828126.914773:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828126.914789:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828126/real 1787828126] req@ffff9e6803e9a940 x1874668790412288/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828126.914805:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790412288/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828126.914816:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828126.914819:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828126.914822:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790412288:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828126.914828:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790412416:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828126.914848:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828126.914858:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828126/real 1787828126] req@ffff9e6803e9a580 x1874668790412416/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828126.914867:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790412416/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828126.914875:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828126.914878:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828126.914881:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790412416:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828126.914886:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790412544:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828126.914899:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828126.914907:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828126/real 1787828126] req@ffff9e6803e9a1c0 x1874668790412544/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828126.914916:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790412544/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828126.914922:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828126.914924:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828126.914927:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790412544:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828126.914931:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790412672:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828126.914943:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828126.914951:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828126/real 1787828126] req@ffff9e6803e9b480 x1874668790412672/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828126.914959:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790412672/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828126.914965:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828126.914968:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828126.914970:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790412672:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828126.914987:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790412800:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828126.915001:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828126.915009:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828126/real 1787828126] req@ffff9e6803e98780 x1874668790412800/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828126.915018:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790412800/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828126.915025:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828126.915027:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828126.915029:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790412800:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828127.425825:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828127.618131:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828128.449805:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828128.642124:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828129.473885:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828129.666147:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828130.497853:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828130.690072:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828131.521864:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828131.713864:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828132.033937:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828132.033948:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828132.033953:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828132.033963:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4768 00000100:00080000:3.0:1787828132.033974:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828132.033992:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828132.034033:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790412928/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828132.034091:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828132.034097:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828132.034101:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828132.034106:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4768 00000100:00080000:3.0:1787828132.034111:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828132.034116:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828132.034128:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828132.034144:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790413056/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828132.034159:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828132.034163:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828132.034166:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828132.034170:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4768 00000100:00080000:3.0:1787828132.034174:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828132.034183:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828132.034185:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790412928:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787828132.034189:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828132.034207:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790413184/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828132.034222:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828132.034226:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828132.034228:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828132.034245:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4768 00000100:00080000:3.0:1787828132.034250:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828132.034254:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828132.034261:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828132.034275:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790413312/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828132.034289:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787828132.034290:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828132.034300:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828132.034304:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828132.034311:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4768 00000100:00080000:3.0:1787828132.034317:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828132.034322:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828132.034329:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828132.034344:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790413440/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828132.034359:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828132.034378:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828132/real 1787828132] req@ffff9e680a2d5a40 x1874668790412928/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828132.034388:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828132.034394:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828132.034409:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4768 00000100:00080000:3.0:1787828132.034417:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828132.034428:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790412928/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828132.034428:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828132.034444:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828132.034469:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790413568/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828132.034471:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828132.034506:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828132.034524:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828132.034525:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828132.034535:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828132.034543:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790412928:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828132.034547:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4768 00000100:00080000:3.0:1787828132.034560:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828132.034568:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828132.034575:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790413056:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787828132.034577:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828132.034603:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790413696/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828132.034617:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828132.034622:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828132.034625:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828132.034629:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4768 00000100:00080000:3.0:1787828132.034634:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828132.034644:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1787828132.034649:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787828132.034652:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828132.034671:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790413824/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828132.034684:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828132.034772:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828132.034775:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828132.034781:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4768 00000100:00080000:3.0:1787828132.034785:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828132.034799:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828132.034805:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828132.034810:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828132/real 1787828132] req@ffff9e680a2d4000 x1874668790413056/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828132.034822:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790413952/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828132.034887:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790413056/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828132.034931:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828132.034940:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828132.034948:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790413056:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828132.035003:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790413184:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828132.035039:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828132.035057:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828132/real 1787828132] req@ffff9e680a2d70c0 x1874668790413184/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828132.035072:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790413184/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828132.035083:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828132.035087:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828132.035091:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790413184:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828132.035098:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790413312:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828132.035118:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828132.035128:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828132/real 1787828132] req@ffff9e680a2d6580 x1874668790413312/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828132.035142:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790413312/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828132.035152:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828132.035155:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828132.035160:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790413312:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828132.035167:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790413440:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828132.035185:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828132.035196:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828132/real 1787828132] req@ffff9e680a2d61c0 x1874668790413440/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828132.035208:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790413440/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828132.035218:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828132.035221:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828132.035225:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790413440:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828132.035231:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790413568:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828132.035260:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828132.035271:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828132/real 1787828132] req@ffff9e680a2d43c0 x1874668790413568/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828132.035285:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790413568/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828132.035296:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828132.035299:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828132.035303:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790413568:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828132.035309:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790413696:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828132.035327:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828132.035339:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828132/real 1787828132] req@ffff9e680a2d4b40 x1874668790413696/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828132.035399:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790413696/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828132.035408:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828132.035412:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828132.035415:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790413696:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828132.035423:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790413824:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828132.035450:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828132.035483:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828132/real 1787828132] req@ffff9e680a2d52c0 x1874668790413824/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828132.035497:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790413824/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828132.035507:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828132.035510:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828132.035514:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790413824:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828132.035520:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790413952:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828132.035540:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828132.035551:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828132/real 1787828132] req@ffff9e680a2d4f00 x1874668790413952/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828132.035564:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790413952/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828132.035574:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828132.035578:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828132.035582:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790413952:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828132.545833:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828132.738125:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828133.569887:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828133.763894:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828134.593799:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828134.786061:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828135.617882:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828135.810003:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828136.641841:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828136.833892:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828137.153858:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828137.153872:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828137.153878:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828137.153891:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4773 00000100:00080000:1.0:1787828137.153904:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828137.153914:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828137.153963:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790414080/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828137.154004:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828137.154009:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828137.154011:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828137.154016:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4773 00000100:00080000:1.0:1787828137.154021:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828137.154025:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828137.154037:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828137.154056:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790414208/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828137.154071:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828137.154080:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828137.154083:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828137.154089:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4773 00000100:00080000:1.0:1787828137.154094:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828137.154098:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828137.154104:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828137.154120:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790414080:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828137.154137:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790414336/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828137.154205:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828137.154211:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828137.154214:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828137.154222:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4773 00000100:00080000:1.0:1787828137.154230:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828137.154238:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828137.154246:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828137.154272:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790414464/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828137.154283:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828137.154288:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828137.154291:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828137.154308:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828137.154310:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4773 00000100:00080000:1.0:1787828137.154325:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828137.154330:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828137.154336:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828137.154351:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790414592/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828137.154362:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828137.154366:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828137.154367:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828137.154371:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4773 00000100:00080000:1.0:1787828137.154374:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828137.154378:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828137.154383:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828137.154397:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790414720/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828137.154405:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828137.154408:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828137.154410:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828137.154414:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4773 00000100:00080000:1.0:1787828137.154418:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828137.154423:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828137.154430:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828137.154457:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790414848/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828137.154458:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828137/real 1787828137] req@ffff9e6831aa8000 x1874668790414080/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828137.154510:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828137.154532:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828137.154535:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828137.154546:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4773 00000100:00080000:1.0:1787828137.154550:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828137.154562:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828137.154563:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790414080/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828137.154582:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828137.154603:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828137.154606:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790414976/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828137.154624:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828137.154646:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828137.154648:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790414080:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828137.154654:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828137.154750:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790414208:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828137.154796:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828137.154822:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828137/real 1787828137] req@ffff9e6831aa9a40 x1874668790414208/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828137.154847:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790414208/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828137.154857:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828137.154878:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4773 00000100:00080000:2.0:1787828137.154879:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828137.154893:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828137.154894:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828137.154923:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790414208:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828137.154932:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828137.154945:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828137.154965:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790415104/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828137.154978:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790414336:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828137.155028:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828137.155044:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828137/real 1787828137] req@ffff9e6831aab480 x1874668790414336/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828137.155059:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790414336/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828137.155070:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828137.155074:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828137.155079:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790414336:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828137.155087:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790414464:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828137.155119:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828137.155131:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828137/real 1787828137] req@ffff9e6831aabc00 x1874668790414464/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828137.155145:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790414464/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828137.155155:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828137.155158:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828137.155162:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790414464:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828137.155169:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790414592:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828137.155188:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828137.155199:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828137/real 1787828137] req@ffff9e6831aaad00 x1874668790414592/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828137.155212:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790414592/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828137.155222:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828137.155226:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828137.155229:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790414592:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828137.155238:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790414720:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828137.155275:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828137.155297:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828137/real 1787828137] req@ffff9e6831aaa1c0 x1874668790414720/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828137.155338:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790414720/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828137.155360:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828137.155366:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828137.155372:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790414720:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828137.155380:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790414848:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828137.155413:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828137.155429:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828137/real 1787828137] req@ffff9e6831aa8780 x1874668790414848/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828137.155444:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790414848/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828137.155454:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828137.155467:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828137.155471:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790414848:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828137.155478:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790414976:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828137.155499:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828137.155510:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828137/real 1787828137] req@ffff9e6831aaa940 x1874668790414976/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828137.155524:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790414976/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828137.155543:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828137.155547:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828137.155551:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790414976:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828137.155576:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790415104:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828137.155598:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828137.155610:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828137/real 1787828137] req@ffff9e6831aa9e00 x1874668790415104/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828137.155624:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790415104/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828137.155634:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828137.155638:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828137.155641:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790415104:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828137.665842:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828137.857890:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828138.689896:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828138.881986:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828139.713888:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828139.905846:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828140.737831:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828140.929831:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828141.761774:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828141.953919:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828142.273837:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828142.273848:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828142.273852:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828142.273863:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4778 00000100:00080000:3.0:1787828142.273874:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828142.273881:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828142.273920:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790415232/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828142.273950:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828142.273955:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828142.273957:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828142.273961:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4778 00000100:00080000:3.0:1787828142.273964:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828142.273969:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828142.273990:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828142.274002:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790415360/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828142.274012:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828142.274016:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828142.274018:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828142.274021:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4778 00000100:00080000:3.0:1787828142.274024:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828142.274027:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828142.274031:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828142.274044:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790415232:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828142.274044:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790415488/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828142.274067:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828142.274070:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828142.274072:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828142.274075:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4778 00000100:00080000:3.0:1787828142.274080:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828142.274083:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828142.274087:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828142.274099:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790415616/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828142.274107:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828142.274111:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828142.274114:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828142.274117:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4778 00000100:00080000:3.0:1787828142.274121:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828142.274124:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828142.274128:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828142.274139:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828142.274143:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790415744/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828142.274153:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828142.274156:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828142.274158:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828142.274161:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4778 00000100:00080000:3.0:1787828142.274164:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828142.274167:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828142.274171:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828142.274179:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790415872/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828142.274186:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828142.274189:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828142.274190:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828142.274194:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4778 00000100:00080000:3.0:1787828142.274198:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828142.274203:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828142.274208:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828142.274219:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790416000/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828142.274230:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828142.274233:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828142.274235:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828142.274240:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828142/real 1787828142] req@ffff9e680a2d5e00 x1874668790415232/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828142.274244:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4778 00000100:00080000:3.0:1787828142.274252:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828142.274263:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828142.274271:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828142.274279:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790415232/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828142.274285:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790416128/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828142.274311:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828142.274313:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828142.274323:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828142.274324:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828142.274328:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828142.274345:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790415232:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828142.274353:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4778 00000100:00100000:2.0:1787828142.274361:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790415360:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828142.274364:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828142.274375:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828142.274385:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828142.274392:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828142.274419:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790416256/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828142.274421:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828142/real 1787828142] req@ffff9e680a2d7480 x1874668790415360/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828142.274500:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790415360/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828142.274513:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828142.274519:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828142.274525:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790415360:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828142.274556:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790415488:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828142.274578:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828142.274590:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828142/real 1787828142] req@ffff9e680a2d7c00 x1874668790415488/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828142.274603:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790415488/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828142.274613:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828142.274616:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828142.274663:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790415488:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828142.274683:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790415616:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828142.274739:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828142.274753:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828142/real 1787828142] req@ffff9e680a2d6d00 x1874668790415616/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828142.274765:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790415616/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828142.274774:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828142.274776:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828142.274780:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790415616:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828142.274785:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790415744:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828142.274801:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828142.274809:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828142/real 1787828142] req@ffff9e680a2d4780 x1874668790415744/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828142.274820:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790415744/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828142.274827:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828142.274830:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828142.274833:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790415744:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828142.274837:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790415872:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828142.274851:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828142.274860:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828142/real 1787828142] req@ffff9e680a2d4f00 x1874668790415872/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828142.274870:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790415872/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828142.274886:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828142.274891:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828142.274896:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790415872:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828142.274904:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790416000:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828142.274933:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828142.274950:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828142/real 1787828142] req@ffff9e680a2d52c0 x1874668790416000/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828142.274996:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790416000/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828142.275009:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828142.275013:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828142.275018:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790416000:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828142.275029:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790416128:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828142.275067:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828142.275089:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828142/real 1787828142] req@ffff9e680a2d4b40 x1874668790416128/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828142.275112:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790416128/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828142.275124:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828142.275127:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828142.275132:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790416128:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828142.275139:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790416256:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828142.275165:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828142.275178:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828142/real 1787828142] req@ffff9e680a2d43c0 x1874668790416256/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828142.275191:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790416256/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828142.275200:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828142.275204:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828142.275207:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790416256:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828142.785826:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828142.977904:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828143.809793:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828144.001942:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828144.833857:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828145.025876:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828145.857783:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828146.049836:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828146.881810:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828147.073876:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828147.393813:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828147.393828:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828147.393834:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828147.393848:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4784 00000100:00080000:3.0:1787828147.393863:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828147.393873:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828147.393921:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790416384/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828147.393956:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828147.393960:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828147.393962:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828147.393967:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4784 00000100:00080000:3.0:1787828147.393972:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828147.393977:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828147.394005:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828147.394019:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790416384:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828147.394029:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790416512/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828147.394046:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828147.394051:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828147.394054:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828147.394058:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4784 00000100:00080000:3.0:1787828147.394063:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828147.394070:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828147.394077:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828147.394091:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790416640/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828147.394104:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828147.394109:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1787828147.394116:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828147.394117:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828147.394130:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4784 00000100:00080000:3.0:1787828147.394138:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828147.394144:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828147.394151:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828147.394166:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790416768/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828147.394196:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828147.394203:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828147/real 1787828147] req@ffff9e680a2d61c0 x1874668790416384/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828147.394203:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828147.394208:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828147.394216:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4784 00000100:00080000:3.0:1787828147.394222:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828147.394231:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828147.394240:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828147.394262:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790416384/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828147.394262:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790416896/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828147.394310:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828147.394315:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828147.394316:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828147.394325:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828147.394330:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828147.394335:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4784 00000100:00080000:3.0:1787828147.394342:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828147.394343:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790416384:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828147.394353:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828147.394360:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828147.394372:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790417024/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828147.394382:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790416512:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828147.394391:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828147.394399:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828147.394402:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828147.394412:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4784 00000100:00080000:3.0:1787828147.394420:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828147.394421:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828147.394436:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828147.394445:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828147/real 1787828147] req@ffff9e680a2d6580 x1874668790416512/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828147.394447:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828147.394495:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790416512/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828147.394504:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790417152/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828147.394523:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828147.394533:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828147.394537:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828147.394543:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787828147.394551:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790416512:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828147.394553:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828147.394570:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4784 00000100:00100000:2.0:1787828147.394577:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790416640:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828147.394581:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828147.394621:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828147.394648:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828147.394665:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790417280/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828147.394670:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828147.394685:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828147.394692:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828147/real 1787828147] req@ffff9e680a2d70c0 x1874668790416640/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828147.394693:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828147.394699:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828147.394746:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790416640/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828147.394751:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4784 00000100:00080000:3.0:1787828147.394761:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828147.394764:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828147.394771:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787828147.394775:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:3.0:1787828147.394782:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828147.394783:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790416640:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828147.394799:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790416768:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828147.394808:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790417408/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828147.394828:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828147.394843:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828147/real 1787828147] req@ffff9e680a2d4000 x1874668790416768/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828147.394877:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790416768/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828147.394890:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828147.394894:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828147.394898:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790416768:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828147.394904:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790416896:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828147.394925:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828147.394936:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828147/real 1787828147] req@ffff9e680a2d5a40 x1874668790416896/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828147.394949:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790416896/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828147.394960:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828147.394963:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828147.394967:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790416896:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828147.394991:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790417024:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828147.395011:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828147.395023:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828147/real 1787828147] req@ffff9e680a2d43c0 x1874668790417024/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828147.395037:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790417024/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828147.395049:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828147.395052:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828147.395057:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790417024:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828147.395064:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790417152:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828147.395082:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828147.395092:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828147/real 1787828147] req@ffff9e680a2d4b40 x1874668790417152/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828147.395106:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790417152/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828147.395116:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828147.395119:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828147.395123:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790417152:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828147.395130:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790417280:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828147.395147:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828147.395158:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828147/real 1787828147] req@ffff9e680a2d52c0 x1874668790417280/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828147.395187:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790417280/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828147.395196:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828147.395200:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828147.395205:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790417280:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828147.395215:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790417408:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828147.395269:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828147.395291:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828147/real 1787828147] req@ffff9e680a2d4f00 x1874668790417408/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828147.395305:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790417408/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828147.395313:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828147.395316:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828147.395319:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790417408:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828147.905798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828148.097848:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828148.929798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828149.122000:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828149.953807:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828150.145980:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828150.977868:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828151.169949:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828152.001836:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828152.193982:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828152.513832:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828152.513850:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828152.513857:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828152.513873:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4789 00000100:00080000:3.0:1787828152.513901:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828152.513911:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828152.513959:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790417536/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828152.513997:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828152.514005:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828152.514009:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828152.514017:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4789 00000100:00080000:3.0:1787828152.514024:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828152.514032:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828152.514048:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828152.514053:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790417536:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828152.514071:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790417664/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828152.514090:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828152.514096:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828152.514099:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828152.514105:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4789 00000100:00080000:3.0:1787828152.514125:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828152.514132:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828152.514140:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828152.514141:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828152.514159:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790417792/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828152.514177:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828152.514183:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828152.514186:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828152.514193:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4789 00000100:00080000:3.0:1787828152.514199:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828152.514207:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828152.514216:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828152.514220:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828152/real 1787828152] req@ffff9e680a2d4780 x1874668790417536/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828152.514234:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790417920/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828152.514251:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828152.514252:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790417536/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828152.514258:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828152.514263:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828152.514270:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4789 00000100:00080000:2.0:1787828152.514272:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828152.514280:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828152.514288:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790417536:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828152.514289:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828152.514296:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828152.514304:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828152.514322:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790418048/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828152.514326:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790417664:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828152.514341:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828152.514347:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828152.514351:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828152.514357:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4789 00000100:00080000:3.0:1787828152.514363:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828152.514368:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828152.514381:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828152.514389:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828152.514399:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828152/real 1787828152] req@ffff9e680a2d6d00 x1874668790417664/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828152.514407:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790418176/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828152.514425:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790417664/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828152.514426:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828152.514432:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828152.514437:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828152.514444:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4789 00000100:00080000:3.0:1787828152.514450:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828152.514457:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828152.514457:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787828152.514464:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:3.0:1787828152.514466:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828152.514471:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790417664:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828152.514481:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790418304/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828152.514483:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790417792:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828152.514500:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828152.514506:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828152.514509:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828152.514516:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4789 00000800:00000100:2.0:1787828152.514517:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828152.514523:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828152.514529:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828152.514536:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828152/real 1787828152] req@ffff9e680a2d7c00 x1874668790417792/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828152.514538:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828152.514589:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790418432/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828152.514592:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790417792/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828152.514619:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828152.514622:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828152.514626:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828152.514629:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828152.514630:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828152.514636:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790417792:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828152.514636:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4789 00000100:00080000:3.0:1787828152.514643:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828152.514649:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790417920:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787828152.514650:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828152.514659:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828152.514677:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790418560/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828152.514687:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828152.514754:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828152/real 1787828152] req@ffff9e680a2d7480 x1874668790417920/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828152.514782:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790417920/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828152.514799:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828152.514806:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828152.514814:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790417920:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828152.514840:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790418048:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828152.514871:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828152.514886:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828152/real 1787828152] req@ffff9e680a2d5e00 x1874668790418048/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828152.514901:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790418048/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828152.514912:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828152.514916:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828152.514919:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790418048:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828152.514926:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790418176:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828152.514945:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828152.514955:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828152/real 1787828152] req@ffff9e680a2d4780 x1874668790418176/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828152.514969:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790418176/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828152.514981:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828152.514984:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828152.514987:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790418176:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828152.514993:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790418304:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828152.515012:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828152.515023:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828152/real 1787828152] req@ffff9e680a2d4f00 x1874668790418304/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828152.515037:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790418304/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828152.515046:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828152.515049:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828152.515053:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790418304:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828152.515059:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790418432:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828152.515077:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828152.515087:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828152/real 1787828152] req@ffff9e680a2d52c0 x1874668790418432/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828152.515100:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790418432/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828152.515109:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828152.515113:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828152.515117:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790418432:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828152.515123:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790418560:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828152.515141:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828152.515151:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828152/real 1787828152] req@ffff9e680a2d4b40 x1874668790418560/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828152.515164:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790418560/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828152.515174:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828152.515178:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828152.515182:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790418560:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828153.025813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828153.217922:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828154.049860:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828154.241989:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828155.073807:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828155.265895:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828156.097827:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828156.289923:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828157.121800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828157.313965:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828157.633899:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828157.633912:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828157.633918:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828157.633935:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4794 00000100:00080000:1.0:1787828157.633949:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828157.633959:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828157.634007:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790418688/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828157.634050:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828157.634057:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828157.634060:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828157.634065:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4794 00000100:00080000:1.0:1787828157.634070:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828157.634076:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828157.634089:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828157.634112:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790418816/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828157.634132:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828157.634133:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790418688:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828157.634144:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828157.634151:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828157.634158:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4794 00000100:00080000:1.0:1787828157.634164:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828157.634169:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828157.634177:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828157.634201:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790418944/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828157.634222:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828157.634227:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828157.634231:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828157.634238:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4794 00000100:00080000:1.0:1787828157.634245:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787828157.634256:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828157.634257:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828157.634272:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828157.634292:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790419072/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828157.634307:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828157.634312:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828157.634316:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828157.634324:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4794 00000100:00080000:1.0:1787828157.634332:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828157.634336:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828157.634350:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828157.634366:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790419200/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828157.634377:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828157.634382:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828157.634384:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828157.634389:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4794 00000100:00080000:1.0:1787828157.634392:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828157.634399:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828157.634405:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828157.634418:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790419328/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828157.634437:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828157.634441:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828157.634444:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828157.634449:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4794 00000100:00080000:1.0:1787828157.634453:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828157.634458:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828157.634463:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828157.634478:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790419456/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828157.634490:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828157.634494:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828157.634497:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828157.634500:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4794 00000100:00080000:1.0:1787828157.634504:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828157.634508:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828157.634539:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828157.634557:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790419584/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828157.634568:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828157.634571:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828157.634573:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828157.634578:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4794 00000100:00080000:1.0:1787828157.634583:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828157.634587:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828157.634592:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828157.634606:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790419712/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828157.634691:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828157/real 1787828157] req@ffff9e6831aaa580 x1874668790418688/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828157.634839:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790418688/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828157.634856:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828157.634862:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828157.634868:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790418688:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828157.634898:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790418816:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828157.634940:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828157.634967:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828157/real 1787828157] req@ffff9e6831aa9680 x1874668790418816/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828157.634995:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790418816/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828157.635020:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828157.635027:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828157.635035:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790418816:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828157.635044:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790418944:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828157.635075:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828157.635091:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828157/real 1787828157] req@ffff9e6831aa8b40 x1874668790418944/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828157.635106:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790418944/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828157.635117:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828157.635121:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828157.635125:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790418944:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828157.635132:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790419072:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828157.635151:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828157.635162:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828157/real 1787828157] req@ffff9e6831aa8f00 x1874668790419072/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828157.635176:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790419072/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828157.635186:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828157.635190:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828157.635194:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790419072:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828157.635201:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790419200:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828157.635219:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828157.635230:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828157/real 1787828157] req@ffff9e6831aa9e00 x1874668790419200/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828157.635245:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790419200/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828157.635254:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828157.635258:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828157.635268:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790419200:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828157.635274:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790419328:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828157.635293:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828157.635303:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828157/real 1787828157] req@ffff9e6831aaa940 x1874668790419328/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828157.635318:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790419328/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828157.635330:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828157.635333:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828157.635337:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790419328:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828157.635343:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790419456:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828157.635371:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828157.635382:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828157/real 1787828157] req@ffff9e6831aa8780 x1874668790419456/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828157.635396:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790419456/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828157.635406:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828157.635409:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828157.635414:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790419456:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828157.635420:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790419584:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828157.635441:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828157.635452:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828157/real 1787828157] req@ffff9e6831aaa1c0 x1874668790419584/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828157.635464:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790419584/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828157.635474:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828157.635478:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828157.635481:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790419584:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828157.635488:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790419712:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828157.635506:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828157.635542:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828157/real 1787828157] req@ffff9e6831aaad00 x1874668790419712/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828157.635558:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790419712/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828157.635576:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828157.635580:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828157.635585:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790419712:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828158.145788:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828158.338000:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828159.169797:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828159.361930:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828160.193846:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828160.385943:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828161.217794:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828161.409958:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828162.241826:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828162.433930:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828162.753865:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828162.753881:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828162.753889:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828162.753905:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4799 00000100:00080000:2.0:1787828162.753921:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828162.753934:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828162.753991:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790419840/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828162.754042:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828162.754049:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828162.754053:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828162.754077:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4799 00000100:00080000:2.0:1787828162.754087:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828162.754098:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790419840:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828162.754104:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828162.754125:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828162.754158:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790419968/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828162.754182:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787828162.754194:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828162.754195:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828162.754203:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828162.754215:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4799 00000100:00080000:2.0:1787828162.754226:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828162.754235:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828162.754246:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828162.754253:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828162/real 1787828162] req@ffff9e6804420780 x1874668790419840/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828162.754279:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790420096/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828162.754285:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790419840/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828162.754337:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828162.754342:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828162.754345:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828162.754357:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790419840:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828162.754359:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828162.754370:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828162.754383:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4799 00000100:00080000:2.0:1787828162.754393:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828162.754399:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790419968:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828162.754410:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828162.754424:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787828162.754438:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828162.754458:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790420224/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828162.754462:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828162/real 1787828162] req@ffff9e6804422940 x1874668790419968/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828162.754524:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790419968/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828162.754537:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828162.754541:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828162.754554:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828162.754555:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787828162.754574:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828162.754575:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790419968:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828162.754596:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4799 00000100:00100000:3.0:1787828162.754598:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790420096:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828162.754612:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828162.754626:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:3.0:1787828162.754635:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:2.0:1787828162.754638:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828162.754653:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828162/real 1787828162] req@ffff9e6804421e00 x1874668790420096/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828162.754681:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790420096/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828162.754684:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790420352/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828162.754762:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828162.754765:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828162.754772:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787828162.754781:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1787828162.754783:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790420096:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828162.754789:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828162.754805:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4799 00000100:00100000:3.0:1787828162.754812:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790420224:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787828162.754816:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828162.754832:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828162.754845:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828162.754851:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828162.754874:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828162/real 1787828162] req@ffff9e6804422d00 x1874668790420224/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828162.754885:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790420480/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828162.754897:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790420224/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828162.754939:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828162.754941:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828162.754947:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787828162.754957:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787828162.754959:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790420224:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787828162.754965:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828162.754973:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790420352:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787828162.754982:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4799 00000100:00080000:2.0:1787828162.754997:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787828162.755003:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828162.755014:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828162.755023:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828162/real 1787828162] req@ffff9e6804420f00 x1874668790420352/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828162.755031:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828162.755044:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790420352/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828162.755058:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787828162.755067:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790420608/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828162.755068:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828162.755078:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790420352:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828162.755105:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790420480:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828162.755107:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828162.755120:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828162.755125:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828162.755138:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4799 00000800:00000100:3.0:1787828162.755140:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828162.755154:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828162.755164:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828162/real 1787828162] req@ffff9e6804421e00 x1874668790420480/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828162.755173:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828162.755195:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828162.755200:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790420480/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828162.755217:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828162.755222:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828162.755232:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790420736/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828162.755233:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790420480:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828162.755259:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790420608:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828162.755290:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787828162.755299:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828162.755305:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828162.755312:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828162.755320:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828162/real 1787828162] req@ffff9e6804422940 x1874668790420608/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828162.755326:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4799 00000100:00080000:2.0:1787828162.755344:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828162.755353:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790420608/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828162.755363:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828162.755383:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787828162.755386:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828162.755399:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828162.755404:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790420608:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828162.755422:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790420864/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828162.755425:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790420736:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828162.755461:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828162.755501:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828162/real 1787828162] req@ffff9e6804420780 x1874668790420736/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828162.755519:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790420736/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828162.755539:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828162.755543:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828162.755547:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790420736:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828162.755560:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790420864:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828162.755583:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828162.755595:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828162/real 1787828162] req@ffff9e6804421680 x1874668790420864/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828162.755608:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790420864/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828162.755618:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828162.755621:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828162.755626:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790420864:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828163.265864:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828163.457984:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828164.289857:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828164.482121:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828165.313876:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828165.506050:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828166.337929:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828166.529961:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828167.361871:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828167.553907:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828167.873840:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828167.873854:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828167.873861:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828167.873874:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4804 00000100:00080000:2.0:1787828167.873887:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828167.873896:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828167.873943:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790420992/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828167.873981:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828167.873987:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828167.873989:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828167.873994:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4804 00000100:00080000:2.0:1787828167.873998:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828167.874003:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828167.874029:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828167.874041:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790421120/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828167.874051:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790420992:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828167.874052:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828167.874059:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828167.874062:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828167.874066:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4804 00000100:00080000:2.0:1787828167.874069:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828167.874071:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828167.874075:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828167.874100:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790421248/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828167.874114:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828167.874119:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828167.874121:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828167.874126:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4804 00000100:00080000:2.0:1787828167.874132:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828167.874136:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828167.874144:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828167.874158:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790421376/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828167.874169:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828167.874173:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828167.874177:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828167.874183:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4804 00000100:00080000:2.0:1787828167.874187:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828167.874195:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828167.874202:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828167.874215:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790421120:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828167.874224:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790421504/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828167.874241:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828167.874245:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828167.874248:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828167.874255:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4804 00000100:00080000:2.0:1787828167.874279:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828167.874282:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828167.874287:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828167.874294:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790421248:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828167.874300:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790421632/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828167.874310:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828167.874312:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828167.874315:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828167.874318:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4804 00000100:00080000:2.0:1787828167.874321:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828167.874324:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828167.874327:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828167.874333:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790421376:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828167.874337:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790421760/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828167.874350:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828167.874353:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828167.874355:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828167.874358:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4804 00000100:00080000:2.0:1787828167.874361:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828167.874364:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828167.874369:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828167.874375:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790421504:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828167.874385:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790421888/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828167.874394:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790421632:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828167.874404:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828167.874409:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828167.874413:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828167.874420:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790421760:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828167.874422:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4804 00000100:00080000:2.0:1787828167.874431:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828167.874436:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828167.874454:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828167.874460:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790421888:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828167.874468:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790422016/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828167.874489:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790422016:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828168.385840:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828168.577937:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828169.409766:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828169.601978:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828170.433893:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828170.625911:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787828170.946213:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:3.0:1787828170.946294:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787828170.946321:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787828170.946416:0:2386:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:3.0:1787828170.946467:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787828170.946485:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787828170.946498:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787828170.946506:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828167/real 1787828170] req@ffff9e6804420780 x1874668790422016/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787828170.946512:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787828170.946526:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787828170.946539:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787828170.946550:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790422016/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787828170.946552:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00080000:2.0:1787828170.946582:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828170.946592:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828170.946601:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790422016:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828170.946616:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828167/real 1787828170] req@ffff9e6804421680 x1874668790421888/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828170.946639:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790421888/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828170.946650:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828170.946656:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828170.946663:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790421888:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828170.946676:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828167/real 1787828170] req@ffff9e6804422580 x1874668790421504/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828170.946764:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790421504/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828170.946784:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828170.946793:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828170.946799:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790421504:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828170.946811:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828167/real 1787828170] req@ffff9e6804420b40 x1874668790421632/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828170.946844:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790421632/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828170.946863:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828170.946874:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828170.946880:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790421632:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828170.946902:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828167/real 1787828170] req@ffff9e6804423480 x1874668790421760/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828170.946924:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790421760/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828170.946938:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828170.946945:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828170.946951:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790421760:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828170.946961:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828167/real 1787828170] req@ffff9e6804423840 x1874668790421120/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828170.946982:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790421120/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828170.946998:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828170.947005:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828170.947010:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790421120:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828170.947021:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828167/real 1787828170] req@ffff9e68044221c0 x1874668790421248/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828170.947044:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790421248/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828170.947061:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828170.947068:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828170.947073:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790421248:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828170.947086:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828167/real 1787828170] req@ffff9e68044230c0 x1874668790421376/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828170.947109:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790421376/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828170.947124:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828170.947130:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828170.947137:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790421376:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828170.947176:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828167/real 1787828170] req@ffff9e68044212c0 x1874668790420992/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828170.947202:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790420992/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828170.947218:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828170.947225:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828170.947232:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790420992:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828171.457812:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828171.650033:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828172.481863:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828172.673984:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828172.993846:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828172.993863:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828172.993870:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828172.993886:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4809 00000100:00080000:2.0:1787828172.993903:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828172.993917:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828172.993968:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790422144/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828172.994014:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828172.994021:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828172.994025:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828172.994033:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4809 00000100:00080000:2.0:1787828172.994042:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828172.994050:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828172.994066:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828172.994092:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790422272/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828172.994113:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828172.994118:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828172.994122:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828172.994141:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4809 00000100:00100000:3.0:1787828172.994149:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790422144:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828172.994152:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828172.994180:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828172.994185:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828172.994199:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790422400/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828172.994217:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828172.994221:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828172.994223:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828172.994227:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4809 00000800:00000100:3.0:1787828172.994232:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828172.994234:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828172.994241:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828172.994253:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828172.994267:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790422528/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828172.994276:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828172/real 1787828172] req@ffff9e68044212c0 x1874668790422144/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828172.994281:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828172.994289:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828172.994293:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828172.994300:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4809 00000100:00080000:2.0:1787828172.994305:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828172.994305:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790422144/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828172.994312:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828172.994320:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787828172.994326:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787828172.994333:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790422656/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828172.994334:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828172.994343:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790422144:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828172.994346:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828172.994351:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828172.994353:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828172.994356:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790422272:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828172.994359:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4809 00000100:00080000:2.0:1787828172.994364:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828172.994368:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828172.994373:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828172.994386:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790422784/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828172.994390:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828172.994401:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828172.994405:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828172.994423:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828172.994429:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828172/real 1787828172] req@ffff9e68044230c0 x1874668790422272/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828172.994430:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4809 00000100:00080000:2.0:1787828172.994438:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828172.994446:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828172.994453:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828172.994456:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790422272/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828172.994468:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787828172.994469:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790422912/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828172.994480:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787828172.994487:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828172.994488:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790422272:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828172.994497:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828172.994500:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828172.994505:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4809 00000100:00080000:2.0:1787828172.994509:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828172.994513:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828172.994518:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828172.994523:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790422400:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828172.994531:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790423040/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828172.994543:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828172.994546:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828172.994548:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787828172.994552:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828172.994557:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4809 00000100:00080000:2.0:1787828172.994562:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828172.994567:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828172.994573:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828172/real 1787828172] req@ffff9e68044221c0 x1874668790422400/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828172.994574:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828172.994592:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790423168/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828172.994596:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790422400/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828172.994617:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828172.994621:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828172.994625:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790422400:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828172.994634:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790422528:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828172.994656:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828172.994667:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828172/real 1787828172] req@ffff9e6804423840 x1874668790422528/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828172.994679:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790422528/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828172.994687:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828172.994690:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828172.994693:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790422528:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828172.994721:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790422656:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828172.994812:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828172.994825:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828172/real 1787828172] req@ffff9e6804423480 x1874668790422656/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828172.994836:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790422656/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828172.994844:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828172.994847:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828172.994851:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790422656:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828172.994857:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790422784:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828172.994878:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828172.994888:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828172/real 1787828172] req@ffff9e6804420b40 x1874668790422784/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828172.994901:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790422784/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828172.994910:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828172.994914:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828172.994917:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790422784:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828172.994923:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790422912:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828172.994943:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828172.994954:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828172/real 1787828172] req@ffff9e6804422580 x1874668790422912/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828172.994973:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790422912/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828172.994982:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828172.994985:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828172.994994:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790422912:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828172.995016:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790423040:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828172.995037:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828172.995049:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828172/real 1787828172] req@ffff9e6804421680 x1874668790423040/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828172.995060:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790423040/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828172.995068:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828172.995072:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828172.995075:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790423040:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828172.995080:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790423168:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828172.995100:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828172.995111:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828172/real 1787828172] req@ffff9e6804420780 x1874668790423168/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828172.995122:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790423168/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828172.995135:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828172.995138:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828172.995141:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790423168:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828173.505809:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828173.697922:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828174.529812:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828174.721902:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828175.553816:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828175.745936:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828176.577853:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828176.769916:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828177.601827:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828177.793969:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828178.113913:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828178.113923:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828178.113945:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828178.113956:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4814 00000100:00080000:1.0:1787828178.113969:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828178.113978:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828178.114040:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790423296/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828178.114072:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828178.114079:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828178.114081:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828178.114085:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4814 00000100:00080000:1.0:1787828178.114089:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828178.114093:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828178.114122:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828178.114145:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790423296:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828178.114146:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790423424/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828178.114173:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828178.114177:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828178.114180:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828178.114185:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4814 00000100:00080000:1.0:1787828178.114189:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828178.114194:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828178.114207:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828178.114234:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790423552/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828178.114246:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828178.114251:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828178.114254:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787828178.114265:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828178.114277:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4814 00000100:00080000:1.0:1787828178.114285:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828178.114290:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828178.114299:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828178.114313:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790423680/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828178.114325:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828178.114330:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828178.114332:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828178.114338:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4814 00000100:00080000:1.0:1787828178.114342:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828178.114346:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828178/real 1787828178] req@ffff9e6831aabc00 x1874668790423296/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828178.114350:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828178.114362:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828178.114412:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790423296/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828178.114445:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828178.114452:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828178.114459:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790423296:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828178.114468:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790423808/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828178.114485:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828178.114489:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828178.114492:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828178.114498:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4814 00000100:00080000:1.0:1787828178.114504:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828178.114505:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790423424:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828178.114520:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828178.114547:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828178.114563:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790423936/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828178.114574:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787828178.114578:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828178.114580:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828178.114585:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828178.114593:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4814 00000100:00080000:1.0:1787828178.114598:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828178.114603:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828178/real 1787828178] req@ffff9e6831aab480 x1874668790423424/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828178.114605:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828178.114625:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828178.114647:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790423424/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828178.114648:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790424064/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828178.114689:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828178.114700:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828178.114705:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828178.114720:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828178.114751:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4814 00000100:00080000:1.0:1787828178.114755:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828178.114758:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828178.114763:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828178.114776:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790424192/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828178.114777:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828178.114796:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828178.114797:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790423424:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828178.114805:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828178.114812:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828178.114820:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790423552:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828178.114822:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4814 00000100:00080000:1.0:1787828178.114835:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828178.114844:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828178.114851:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787828178.114872:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828178.114878:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790424320/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828178.114895:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828178/real 1787828178] req@ffff9e6831aa9a40 x1874668790423552/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828178.114932:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790423552/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828178.114947:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828178.114953:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828178.114958:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790423552:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828178.114979:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790423680:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828178.115011:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828178.115030:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828178/real 1787828178] req@ffff9e6831aa8000 x1874668790423680/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828178.115322:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790423680/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828178.115338:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828178.115345:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828178.115352:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790423680:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828178.115363:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790423808:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828178.115427:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828178.115451:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828178/real 1787828178] req@ffff9e6831aaad00 x1874668790423808/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828178.115475:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790423808/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828178.115489:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828178.115495:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828178.115509:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790423808:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828178.115548:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790423936:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828178.115581:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828178.115597:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828178/real 1787828178] req@ffff9e6831aaa1c0 x1874668790423936/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828178.115614:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790423936/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828178.115628:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828178.115632:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828178.115637:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790423936:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828178.115646:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790424064:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828178.115675:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828178.115692:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828178/real 1787828178] req@ffff9e6831aa8780 x1874668790424064/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828178.115774:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790424064/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828178.115786:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828178.115791:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828178.115797:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790424064:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828178.115807:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790424192:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828178.115843:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828178.115865:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828178/real 1787828178] req@ffff9e6831aaa940 x1874668790424192/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828178.115883:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790424192/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828178.115896:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828178.115901:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828178.115906:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790424192:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828178.115916:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790424320:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828178.115947:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828178.115985:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828178/real 1787828178] req@ffff9e6831aa9e00 x1874668790424320/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828178.116003:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790424320/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828178.116016:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828178.116032:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828178.116036:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790424320:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828178.625797:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828178.817892:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828179.649865:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828179.841963:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828180.673773:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828180.865929:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828181.697790:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828181.889885:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828182.721830:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828182.913957:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828183.233852:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828183.233865:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828183.233893:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828183.233907:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4819 00000100:00080000:1.0:1787828183.233920:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828183.233929:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828183.233977:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790424448/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828183.234012:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828183.234018:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828183.234020:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828183.234025:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4819 00000100:00080000:1.0:1787828183.234030:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828183.234034:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828183.234046:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828183.234062:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790424576/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828183.234087:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828183.234094:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790424448:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828183.234095:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828183.234118:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828183.234125:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4819 00000100:00080000:1.0:1787828183.234128:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828183.234131:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828183.234134:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828183.234143:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790424704/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828183.234148:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828183.234152:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828183.234153:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828183.234156:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4819 00000100:00080000:1.0:1787828183.234159:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828183.234161:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828183.234164:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828183.234193:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790424832/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828183.234206:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828183.234210:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828183.234215:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787828183.234219:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828183.234226:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4819 00000100:00080000:1.0:1787828183.234237:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828183.234244:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828183.234261:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828183.234278:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790424960/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828183.234291:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828183.234297:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828183/real 1787828183] req@ffff9e6831aa8f00 x1874668790424448/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828183.234299:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828183.234309:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828183.234330:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4819 00000100:00080000:1.0:1787828183.234386:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828183.234387:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790424448/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828183.234396:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:3.0:1787828183.234401:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787828183.234404:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787828183.234407:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828183.234413:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790424448:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828183.234425:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790425088/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828183.234465:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828183.234470:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787828183.234476:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790424576:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828183.234478:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828183.234495:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4819 00000100:00080000:1.0:1787828183.234502:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828183.234508:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828183.234516:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828183.234532:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828183.234540:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790425216/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828183.234560:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828183/real 1787828183] req@ffff9e6831aa8b40 x1874668790424576/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828183.234625:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790424576/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828183.234629:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828183.234635:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828183.234639:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828183.234646:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4819 00000100:00080000:1.0:1787828183.234652:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828183.234657:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:3.0:1787828183.234660:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787828183.234662:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787828183.234692:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828183.234701:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790425344/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828183.234704:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790424576:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828183.234730:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790424704:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828183.234785:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828183.234789:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828183.234791:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828183.234796:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4819 00000100:00080000:1.0:1787828183.234800:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828183.234802:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828183.234807:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828183.234827:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790425472/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828183.234844:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828183.234858:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828183/real 1787828183] req@ffff9e6831aa9680 x1874668790424704/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828183.234867:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790424704/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828183.234904:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828183.234908:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828183.234912:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790424704:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828183.234921:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790424832:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828183.234956:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828183.234970:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828183/real 1787828183] req@ffff9e6831aaa580 x1874668790424832/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828183.234996:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790424832/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828183.235012:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828183.235018:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828183.235026:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790424832:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828183.235039:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790424960:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828183.235078:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828183.235107:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828183/real 1787828183] req@ffff9e6831aa9e00 x1874668790424960/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828183.235116:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790424960/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828183.235121:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828183.235123:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828183.235125:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790424960:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828183.235129:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790425088:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828183.235142:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828183.235148:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828183/real 1787828183] req@ffff9e6831aaa940 x1874668790425088/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828183.235186:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790425088/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828183.235197:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828183.235201:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828183.235204:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790425088:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828183.235230:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790425216:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828183.235254:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828183.235274:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828183/real 1787828183] req@ffff9e6831aa8780 x1874668790425216/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828183.235288:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790425216/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828183.235298:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828183.235301:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828183.235304:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790425216:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828183.235311:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790425344:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828183.235330:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828183.235374:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828183/real 1787828183] req@ffff9e6831aaa1c0 x1874668790425344/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828183.235390:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790425344/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828183.235401:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828183.235405:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828183.235408:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790425344:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828183.235415:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790425472:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828183.235441:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828183.235454:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828183/real 1787828183] req@ffff9e6831aaad00 x1874668790425472/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828183.235467:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790425472/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828183.235478:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828183.235482:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828183.235486:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790425472:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828183.745782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828183.937950:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828184.769836:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828184.962021:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828185.793799:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828185.985845:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828186.817867:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828187.009905:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828187.841783:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828188.033828:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828188.353803:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828188.353812:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828188.353815:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828188.353822:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4825 00000100:00080000:0.0:1787828188.353831:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828188.353837:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828188.353869:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790425600/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828188.353891:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828188.353894:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828188.353895:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828188.353898:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4825 00000100:00080000:0.0:1787828188.353900:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828188.353902:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828188.353910:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828188.353919:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790425728/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828188.353926:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828188.353929:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828188.353931:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828188.353934:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790425600:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828188.353936:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4825 00000100:00080000:0.0:1787828188.353943:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828188.353945:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828188.353955:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828188.353964:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790425856/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828188.353971:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828188.353974:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828188.353975:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828188.353978:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4825 00000100:00080000:0.0:1787828188.353981:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828188.353983:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828188.353986:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828188.353995:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790425984/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828188.354003:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828188.354005:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828188.354008:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828188.354010:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828188.354013:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4825 00000100:00080000:0.0:1787828188.354015:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828188.354023:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828188.354026:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828188.354035:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790426112/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828188.354041:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828188.354044:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828188.354046:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828188.354050:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828188/real 1787828188] req@ffff9e6803e9ad00 x1874668790425600/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828188.354051:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4825 00000100:00080000:0.0:1787828188.354056:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828188.354063:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828188.354068:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828188.354075:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790425600/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828188.354078:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790426240/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828188.354102:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828188.354103:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828188.354106:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828188.354111:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828188.354111:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828188.354121:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790425600:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828188.354122:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4825 00000100:00080000:0.0:1787828188.354127:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828188.354131:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828188.354133:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828188.354140:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790426368/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828188.354148:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828188.354150:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790425728:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828188.354152:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828188.354155:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828188.354159:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4825 00000100:00080000:0.0:1787828188.354162:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828188.354164:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828188.354168:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828188.354177:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790426496/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828188.354182:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828188.354191:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828188.354194:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828188.354195:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828188.354199:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828188/real 1787828188] req@ffff9e6803e98b40 x1874668790425728/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828188.354202:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4825 00000100:00080000:0.0:1787828188.354208:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828188.354214:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828188.354218:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828188.354219:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790425728/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828188.354228:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790426624/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828188.354249:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828188.354255:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828188.354258:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790425728:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828188.354265:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790425856:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828188.354277:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828188.354284:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828188/real 1787828188] req@ffff9e6803e9bc00 x1874668790425856/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828188.354290:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790425856/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828188.354296:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828188.354297:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828188.354319:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790425856:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828188.354324:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790425984:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828188.354338:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828188.354346:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828188/real 1787828188] req@ffff9e6803e992c0 x1874668790425984/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828188.354353:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790425984/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828188.354359:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828188.354361:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828188.354363:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790425984:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828188.354367:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790426112:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828188.354377:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828188.354382:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828188/real 1787828188] req@ffff9e6803e98780 x1874668790426112/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828188.354389:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790426112/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828188.354394:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828188.354396:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828188.354398:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790426112:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828188.354401:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790426240:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828188.354411:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828188.354416:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828188/real 1787828188] req@ffff9e6803e9b480 x1874668790426240/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828188.354423:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790426240/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828188.354429:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828188.354431:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828188.354433:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790426240:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828188.354446:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790426368:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828188.354456:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828188.354463:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828188/real 1787828188] req@ffff9e6803e9a1c0 x1874668790426368/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828188.354470:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790426368/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828188.354475:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828188.354477:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828188.354479:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790426368:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828188.354482:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790426496:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828188.354491:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828188.354497:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828188/real 1787828188] req@ffff9e6803e9a580 x1874668790426496/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828188.354504:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790426496/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828188.354509:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828188.354510:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828188.354512:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790426496:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828188.354515:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790426624:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828188.354524:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828188.354530:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828188/real 1787828188] req@ffff9e6803e9a940 x1874668790426624/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828188.354536:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790426624/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828188.354542:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828188.354544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828188.354546:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790426624:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828188.865818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828189.057839:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828189.889830:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828190.082000:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828190.913824:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828191.105823:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828191.937764:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828192.129853:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828192.961800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828193.153846:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828193.473798:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828193.473806:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828193.473809:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828193.473818:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4830 00000100:00080000:0.0:1787828193.473828:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828193.473833:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828193.473875:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790426752/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828193.473897:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828193.473899:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828193.473900:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828193.473903:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4830 00000100:00080000:0.0:1787828193.473906:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828193.473911:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828193.473920:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828193.473938:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790426880/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828193.473943:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790426752:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828193.473973:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828193.473987:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828193.473989:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828193.473996:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4830 00000100:00080000:0.0:1787828193.474001:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828193.474004:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828193.474010:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828193.474023:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790427008/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828193.474031:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787828193.474040:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828193.474044:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828193.474047:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828193.474051:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4830 00000100:00080000:0.0:1787828193.474055:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828193.474058:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828193.474065:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828193.474075:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790427136/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828193.474082:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828193.474085:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828193.474087:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828193.474089:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4830 00000100:00080000:0.0:1787828193.474092:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828193.474094:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828193.474097:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828193.474105:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828193/real 1787828193] req@ffff9e6803e99680 x1874668790426752/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828193.474113:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790427264/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828193.474132:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828193.474132:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790426752/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828193.474151:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828193.474160:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828193.474162:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828193.474172:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787828193.474176:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4830 00000100:00100000:3.0:1787828193.474181:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790426752:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828193.474186:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828193.474195:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828193.474201:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828193.474210:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790426880:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828193.474218:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790427392/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828193.474233:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787828193.474237:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828193.474245:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828193.474248:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828193.474254:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828193/real 1787828193] req@ffff9e6803e99e00 x1874668790426880/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828193.474255:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4830 00000100:00080000:0.0:1787828193.474286:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828193.474292:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790426880/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828193.474297:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828193.474312:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787828193.474336:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828193.474342:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790427520/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828193.474347:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828193.474358:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790426880:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828193.474379:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790427008:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828193.474386:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828193.474389:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828193.474392:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828193.474397:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4830 00000100:00080000:0.0:1787828193.474404:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828193.474408:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828193.474420:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787828193.474423:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828193.474437:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790427648/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828193.474443:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828193/real 1787828193] req@ffff9e6803e98f00 x1874668790427008/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828193.474455:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828193.474468:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828193.474474:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828193.474483:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4830 00000100:00080000:0.0:1787828193.474486:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828193.474488:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790427008/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828193.474493:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828193.474506:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787828193.474508:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828193.474514:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828193.474518:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790427776/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828193.474524:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790427008:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828193.474544:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790427136:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828193.474562:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828193.474575:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828193/real 1787828193] req@ffff9e6803e9b840 x1874668790427136/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828193.474583:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790427136/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828193.474589:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828193.474591:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828193.474593:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790427136:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828193.474598:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790427264:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828193.474609:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828193.474616:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828193/real 1787828193] req@ffff9e6803e9a940 x1874668790427264/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828193.474626:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790427264/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828193.474632:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828193.474634:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828193.474636:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790427264:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828193.474657:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790427392:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828193.474671:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828193.474678:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828193/real 1787828193] req@ffff9e6803e9a580 x1874668790427392/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828193.474687:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790427392/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828193.474715:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828193.474720:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828193.474724:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790427392:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828193.474733:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790427520:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828193.474766:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828193.474779:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828193/real 1787828193] req@ffff9e6803e9a1c0 x1874668790427520/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828193.474789:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790427520/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828193.474862:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828193.474868:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828193.474872:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790427520:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828193.474877:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790427648:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828193.475035:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828193.475061:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828193/real 1787828193] req@ffff9e6803e9b480 x1874668790427648/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828193.475071:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790427648/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828193.475078:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828193.475115:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828193.475118:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790427648:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828193.475124:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790427776:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828193.475140:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828193.475148:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828193/real 1787828193] req@ffff9e6803e98780 x1874668790427776/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828193.475156:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790427776/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828193.475162:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828193.475165:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828193.475167:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790427776:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828193.985783:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828194.177843:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828195.009790:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828195.201962:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828196.033795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828196.225881:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828197.057816:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828197.249883:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828198.081804:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828198.273903:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828198.593791:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828198.593803:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828198.593807:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828198.593817:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4835 00000100:00080000:0.0:1787828198.593829:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828198.593837:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828198.593875:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790427904/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828198.593905:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828198.593914:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828198.593916:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828198.593920:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4835 00000100:00080000:0.0:1787828198.593925:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828198.593929:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828198.593940:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828198.593951:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790427904:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787828198.593958:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790428032/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828198.593972:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828198.593975:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828198.593978:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828198.593982:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4835 00000100:00080000:0.0:1787828198.593986:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828198.593991:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828198.593996:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828198.594010:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790428160/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828198.594010:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828198.594023:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828198.594028:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828198.594030:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828198.594034:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4835 00000100:00080000:0.0:1787828198.594039:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828198.594044:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828198.594050:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828198.594055:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828198/real 1787828198] req@ffff9e6803e992c0 x1874668790427904/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828198.594066:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790428288/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828198.594076:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790427904/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828198.594080:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828198.594089:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828198.594091:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828198.594092:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828198.594099:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787828198.594103:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4835 00000100:00100000:3.0:1787828198.594105:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790427904:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828198.594110:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828198.594117:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828198.594123:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828198.594128:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790428032:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828198.594137:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790428416/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828198.594149:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828198.594150:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828198.594156:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828198.594158:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828198.594163:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828198/real 1787828198] req@ffff9e6803e9bc00 x1874668790428032/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828198.594165:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4835 00000100:00080000:0.0:1787828198.594173:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828198.594180:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790428032/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828198.594182:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828198.594192:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787828198.594201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828198.594205:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828198.594211:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790428032:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828198.594212:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790428544/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828198.594218:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790428160:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828198.594245:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828198.594249:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828198.594251:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828198.594257:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4835 00000800:00000100:3.0:1787828198.594261:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828198.594264:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828198.594272:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828198.594273:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828198/real 1787828198] req@ffff9e6803e98b40 x1874668790428160/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828198.594281:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828198.594286:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790428160/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828198.594294:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828198.594299:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790428672/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828198.594300:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828198.594306:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790428160:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828198.594317:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790428288:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787828198.594322:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828198.594327:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828198.594330:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787828198.594335:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828198.594339:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4835 00000100:00080000:0.0:1787828198.594345:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828198.594346:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828198/real 1787828198] req@ffff9e6803e9ad00 x1874668790428288/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828198.594353:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828198.594361:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790428288/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828198.594362:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787828198.594373:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828198.594377:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828198.594382:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790428800/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828198.594382:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790428288:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787828198.594398:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828198.594402:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790428416:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787828198.594404:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828198.594409:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828198.594415:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4835 00000800:00000100:3.0:1787828198.594420:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828198.594422:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828198.594430:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828198.594431:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828198/real 1787828198] req@ffff9e6803e992c0 x1874668790428416/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828198.594438:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828198.594443:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790428416/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828198.594455:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790428928/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828198.594459:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828198.594467:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828198.594474:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790428416:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828198.594481:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790428544:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828198.594497:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828198.594508:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828198/real 1787828198] req@ffff9e6803e98780 x1874668790428544/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828198.594521:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790428544/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828198.594531:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828198.594535:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828198.594539:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790428544:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828198.594545:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790428672:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828198.594563:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828198.594571:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828198/real 1787828198] req@ffff9e6803e9b480 x1874668790428672/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828198.594579:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790428672/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828198.594584:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828198.594586:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828198.594589:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790428672:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828198.594592:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790428800:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828198.594602:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828198.594608:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828198/real 1787828198] req@ffff9e6803e9a1c0 x1874668790428800/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828198.594615:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790428800/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828198.594619:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828198.594621:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828198.594623:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790428800:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828198.594633:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790428928:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828198.594644:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828198.594649:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828198/real 1787828198] req@ffff9e6803e9a580 x1874668790428928/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828198.594659:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790428928/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828198.594667:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828198.594670:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828198.594674:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790428928:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828199.105818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828199.297915:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828200.129780:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828200.321856:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828201.153832:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828201.345890:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828202.177800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828202.369937:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828203.201763:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828203.393915:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828203.713836:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828203.713844:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828203.713849:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828203.713856:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4840 00000100:00080000:3.0:1787828203.713865:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828203.713870:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828203.713901:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790429056/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828203.713926:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828203.713929:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828203.713930:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828203.713933:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4840 00000100:00080000:3.0:1787828203.713935:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828203.713945:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828203.713953:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828203.713968:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790429184/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828203.713976:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828203.713978:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828203.713980:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828203.713983:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4840 00000100:00080000:3.0:1787828203.713986:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828203.713989:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828203.713992:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828203.714006:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790429312/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828203.714008:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790429056:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828203.714016:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828203.714024:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828203.714026:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828203.714030:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4840 00000100:00080000:3.0:1787828203.714032:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828203.714034:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828203.714037:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828203.714044:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790429440/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828203.714053:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828203.714054:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828203.714055:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828203.714058:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4840 00000100:00080000:3.0:1787828203.714060:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828203.714063:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828203.714067:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828203.714074:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790429568/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828203.714082:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828203.714085:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828203.714086:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828203.714089:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4840 00000100:00080000:3.0:1787828203.714095:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828203.714098:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828203.714101:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828203.714107:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790429696/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828203.714119:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828203.714124:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828203.714128:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828203.714129:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828203.714133:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4840 00000100:00080000:3.0:1787828203.714135:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828203.714139:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828203.714145:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828203.714152:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790429824/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828203.714164:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828203.714167:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828203.714169:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828203.714172:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4840 00000100:00080000:3.0:1787828203.714176:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828203.714181:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828203/real 1787828203] req@ffff9e680a2d43c0 x1874668790429056/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828203.714186:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828203.714213:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828203.714228:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790429952/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828203.714232:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790429056/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828203.714268:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828203.714285:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828203.714292:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828203.714303:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828203.714303:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4840 00000100:00080000:3.0:1787828203.714311:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828203.714318:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828203.714323:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828203.714331:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828203.714335:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790429056:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828203.714348:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790430080/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828203.714352:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790429184:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828203.714390:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828203.714405:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828203/real 1787828203] req@ffff9e680a2d5a40 x1874668790429184/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828203.714414:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790429184/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828203.714421:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828203.714424:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828203.714426:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790429184:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828203.714452:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790429312:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828203.714465:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828203.714471:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828203/real 1787828203] req@ffff9e680a2d4000 x1874668790429312/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828203.714478:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790429312/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828203.714484:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828203.714486:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828203.714487:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790429312:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828203.714492:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790429440:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828203.714504:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828203.714509:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828203/real 1787828203] req@ffff9e680a2d70c0 x1874668790429440/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828203.714517:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790429440/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828203.714523:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828203.714525:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828203.714526:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790429440:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828203.714530:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790429568:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828203.714540:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828203.714546:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828203/real 1787828203] req@ffff9e680a2d6580 x1874668790429568/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828203.714553:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790429568/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828203.714559:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828203.714560:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828203.714562:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790429568:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828203.714566:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790429696:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828203.714576:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828203.714582:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828203/real 1787828203] req@ffff9e680a2d61c0 x1874668790429696/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828203.714590:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790429696/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828203.714596:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828203.714598:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828203.714600:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790429696:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828203.714604:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790429824:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828203.714615:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828203.714620:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828203/real 1787828203] req@ffff9e680a2d4b40 x1874668790429824/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828203.714628:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790429824/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828203.714633:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828203.714634:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828203.714636:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790429824:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828203.714640:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790429952:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828203.714650:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828203.714655:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828203/real 1787828203] req@ffff9e680a2d52c0 x1874668790429952/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828203.714663:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790429952/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828203.714673:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828203.714675:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828203.714677:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790429952:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828203.714681:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790430080:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828203.714691:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828203.714722:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828203/real 1787828203] req@ffff9e680a2d4f00 x1874668790430080/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828203.714731:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790430080/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828203.714736:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828203.714738:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828203.714740:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790430080:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828204.225797:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828204.417918:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828205.249783:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828205.441964:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828206.273794:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828206.465866:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828207.297823:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828207.489935:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828208.321781:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828208.513887:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828208.833789:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828208.833798:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828208.833801:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828208.833809:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4845 00000100:00080000:0.0:1787828208.833818:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828208.833824:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828208.833856:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790430208/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828208.833878:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828208.833881:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828208.833882:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828208.833885:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4845 00000100:00080000:0.0:1787828208.833887:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828208.833889:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828208.833897:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828208.833907:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790430336/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828208.833920:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828208.833922:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828208.833923:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828208.833929:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4845 00000100:00080000:0.0:1787828208.833937:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828208.833938:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790430208:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828208.833944:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828208.833954:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828208.833964:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790430464/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828208.833972:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828208.833975:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828208.833976:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828208.833979:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4845 00000100:00080000:0.0:1787828208.833982:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828208.833985:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828208.833988:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828208.833998:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790430592/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828208.834004:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828208.834006:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828208.834007:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828208.834009:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4845 00000100:00080000:0.0:1787828208.834013:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828208.834015:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828208.834020:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828208.834028:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790430720/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828208.834035:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828208.834046:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828208.834048:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828208.834050:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828208.834055:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4845 00000100:00080000:0.0:1787828208.834060:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828208.834064:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828208.834068:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828208.834078:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790430848/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828208.834087:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828208.834090:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828208.834092:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828208.834095:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4845 00000100:00080000:0.0:1787828208.834098:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828208.834103:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828208/real 1787828208] req@ffff9e6803e9a940 x1874668790430208/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828208.834104:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828208.834113:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828208.834131:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790430976/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828208.834139:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790430208/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828208.834146:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828208.834209:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828208.834214:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828208.834218:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790430208:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787828208.834245:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790430336:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828208.834268:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828208.834278:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828208/real 1787828208] req@ffff9e6803e9b840 x1874668790430336/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828208.834285:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828208.834292:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828208.834298:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790430336/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828208.834299:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4845 00000100:00080000:0.0:1787828208.834311:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828208.834325:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787828208.834325:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787828208.834343:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828208.834343:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828208.834353:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790430336:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828208.834360:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790430464:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828208.834364:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790431104/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828208.834379:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828208.834382:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828208.834383:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828208.834387:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828208.834391:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4845 00000100:00080000:0.0:1787828208.834396:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828208.834400:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828208.834404:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828208.834404:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828208/real 1787828208] req@ffff9e6803e98f00 x1874668790430464/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828208.834420:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790431232/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828208.834424:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790430464/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828208.834483:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828208.834489:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828208.834492:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790430464:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828208.834498:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790430592:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828208.834524:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828208.834531:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828208/real 1787828208] req@ffff9e6803e99e00 x1874668790430592/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828208.834540:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790430592/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828208.834546:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828208.834548:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828208.834550:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790430592:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828208.834554:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790430720:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828208.834572:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828208.834584:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828208/real 1787828208] req@ffff9e6803e99680 x1874668790430720/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828208.834596:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790430720/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828208.834606:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828208.834610:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828208.834613:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790430720:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828208.834620:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790430848:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828208.834645:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828208.834658:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828208/real 1787828208] req@ffff9e6803e9a580 x1874668790430848/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828208.834669:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790430848/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828208.834688:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828208.834692:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828208.834737:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790430848:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828208.834745:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790430976:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828208.834785:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828208.834801:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828208/real 1787828208] req@ffff9e6803e9a1c0 x1874668790430976/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828208.834815:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790430976/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828208.834825:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828208.834829:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828208.834833:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790430976:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828208.834860:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790431104:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828208.834885:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828208.834899:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828208/real 1787828208] req@ffff9e6803e9b480 x1874668790431104/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828208.834913:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790431104/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828208.834923:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828208.834927:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828208.834933:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790431104:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828208.834940:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790431232:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828208.834962:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828208.834974:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828208/real 1787828208] req@ffff9e6803e98780 x1874668790431232/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828208.834988:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790431232/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828208.834997:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828208.835001:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828208.835005:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790431232:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828209.345826:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828209.537838:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828210.369765:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828210.561811:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828211.393789:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828211.585851:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828212.417778:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828212.609834:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828213.441787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828213.633859:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828213.953857:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828213.953869:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828213.953873:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828213.953884:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4850 00000100:00080000:2.0:1787828213.953895:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828213.953904:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828213.953948:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790431360/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828213.953974:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828213.953977:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828213.953979:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828213.953981:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4850 00000100:00080000:2.0:1787828213.953984:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828213.953988:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828213.953996:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828213.954005:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790431488/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828213.954013:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828213.954015:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790431360:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828213.954016:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828213.954019:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828213.954026:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4850 00000100:00080000:2.0:1787828213.954028:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828213.954030:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828213.954033:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828213.954041:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790431616/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828213.954056:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828213.954059:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828213.954060:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828213.954063:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4850 00000100:00080000:2.0:1787828213.954065:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828213.954067:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828213.954070:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828213.954077:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790431744/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828213.954083:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828213.954086:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828213.954087:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828213.954090:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4850 00000100:00080000:2.0:1787828213.954092:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828213.954096:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:3.0:1787828213.954099:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:2.0:1787828213.954100:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828213.954110:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790431872/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828213.954117:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828213.954120:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828213.954134:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828213.954138:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4850 00000100:00080000:2.0:1787828213.954142:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828213.954146:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828213.954150:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828213.954172:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790432000/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828213.954182:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828213/real 1787828213] req@ffff9e6804422940 x1874668790431360/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828213.954184:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828213.954191:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828213.954194:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828213.954202:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4850 00000100:00080000:2.0:1787828213.954208:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828213.954212:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828213.954218:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828213.954219:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790431360/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828213.954232:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790432128/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828213.954236:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828213.954247:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787828213.954249:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828213.954255:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787828213.954256:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790431360:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828213.954258:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828213.954266:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4850 00000100:00080000:2.0:1787828213.954270:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828213.954275:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828213.954280:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828213.954290:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790431488:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828213.954293:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790432256/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828213.954305:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828213.954307:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828213.954308:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828213.954311:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4850 00000100:00080000:2.0:1787828213.954321:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828213.954326:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:3.0:1787828213.954328:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:2.0:1787828213.954330:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828213.954341:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790432384/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828213.954350:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828213/real 1787828213] req@ffff9e6804421e00 x1874668790431488/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828213.954366:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790431488/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828213.954376:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828213.954380:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828213.954384:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790431488:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828213.954391:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790431616:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828213.954414:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828213.954429:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828213/real 1787828213] req@ffff9e6804420f00 x1874668790431616/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828213.954442:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790431616/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828213.954450:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828213.954453:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828213.954457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790431616:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828213.954464:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790431744:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828213.954487:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828213.954500:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828213/real 1787828213] req@ffff9e6804422d00 x1874668790431744/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828213.954514:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790431744/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828213.954522:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828213.954526:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828213.954538:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790431744:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828213.954545:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790431872:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828213.954566:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828213.954577:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828213/real 1787828213] req@ffff9e6804420780 x1874668790431872/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828213.954589:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790431872/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828213.954609:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828213.954612:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828213.954616:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790431872:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828213.954622:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790432000:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828213.954639:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828213.954649:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828213/real 1787828213] req@ffff9e6804421680 x1874668790432000/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828213.954661:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790432000/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828213.954672:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828213.954675:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828213.954678:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790432000:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828213.954684:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790432128:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828213.954729:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828213.954741:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828213/real 1787828213] req@ffff9e6804422580 x1874668790432128/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828213.954753:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790432128/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828213.954760:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828213.954763:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828213.954767:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790432128:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828213.954787:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790432256:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828213.954805:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828213.954814:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828213/real 1787828213] req@ffff9e6804420b40 x1874668790432256/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828213.954825:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790432256/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828213.954833:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828213.954836:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828213.954840:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790432256:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828213.954845:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790432384:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828213.954860:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828213.954870:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828213/real 1787828213] req@ffff9e6804423480 x1874668790432384/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828213.954881:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790432384/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828213.954889:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828213.954892:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828213.954895:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790432384:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828214.465792:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828214.657849:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828215.489839:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828215.681873:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828216.513802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828216.705838:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828217.537943:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828217.729854:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828218.561774:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828218.753845:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828219.073790:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828219.073798:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828219.073803:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828219.073812:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4855 00000100:00080000:3.0:1787828219.073821:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828219.073833:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828219.073865:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790432512/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828219.073886:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828219.073889:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828219.073891:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828219.073893:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4855 00000100:00080000:3.0:1787828219.073896:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828219.073898:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828219.073906:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828219.073914:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790432640/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828219.073922:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828219.073927:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828219.073928:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790432512:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828219.073929:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828219.073934:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4855 00000100:00080000:3.0:1787828219.073939:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828219.073942:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828219.073946:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828219.073954:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790432768/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828219.073960:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828219.073963:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828219.073964:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828219.073967:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4855 00000100:00080000:3.0:1787828219.073969:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828219.073971:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828219.073975:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828219.073981:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790432896/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828219.073987:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828219.073989:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828219.073990:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828219.073992:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4855 00000100:00080000:3.0:1787828219.073995:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828219.074001:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787828219.074005:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787828219.074006:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828219.074017:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790433024/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828219.074024:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828219.074026:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828219.074028:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828219.074031:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4855 00000100:00080000:3.0:1787828219.074034:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828219.074037:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828219.074040:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828219.074054:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790433152/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828219.074055:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828219/real 1787828219] req@ffff9e680a2d4780 x1874668790432512/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828219.074066:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828219.074073:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828219.074077:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828219.074083:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4855 00000100:00080000:3.0:1787828219.074099:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828219.074104:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828219.074107:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828219.074115:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790433280/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828219.074122:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828219.074124:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828219.074126:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828219.074132:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4855 00000100:00100000:2.0:1787828219.074133:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790432512/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828219.074135:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828219.074139:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828219.074144:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828219.074154:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790433408/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828219.074157:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828219.074162:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828219.074166:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828219.074167:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828219.074168:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828219.074176:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4855 00000100:00100000:2.0:1787828219.074179:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790432512:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828219.074180:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828219.074186:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828219.074192:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828219.074198:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790433536/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828219.074215:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790432640:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828219.074242:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828219.074256:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828219/real 1787828219] req@ffff9e680a2d5e00 x1874668790432640/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828219.074269:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790432640/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828219.074278:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828219.074282:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828219.074286:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790432640:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828219.074292:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790432768:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828219.074313:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828219.074325:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828219/real 1787828219] req@ffff9e680a2d7480 x1874668790432768/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828219.074336:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790432768/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828219.074352:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828219.074355:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828219.074359:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790432768:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828219.074366:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790432896:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828219.074387:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828219.074398:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828219/real 1787828219] req@ffff9e680a2d7c00 x1874668790432896/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828219.074411:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790432896/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828219.074420:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828219.074423:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828219.074426:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790432896:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828219.074433:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790433024:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828219.074452:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828219.074463:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828219/real 1787828219] req@ffff9e680a2d6d00 x1874668790433024/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828219.074475:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790433024/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828219.074484:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828219.074487:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828219.074491:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790433024:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828219.074497:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790433152:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828219.074517:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828219.074528:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828219/real 1787828219] req@ffff9e680a2d4f00 x1874668790433152/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828219.074541:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790433152/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828219.074551:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828219.074554:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828219.074557:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790433152:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828219.074563:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790433280:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828219.074580:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828219.074592:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828219/real 1787828219] req@ffff9e680a2d52c0 x1874668790433280/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828219.074606:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790433280/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828219.074615:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828219.074618:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828219.074622:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790433280:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828219.074628:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790433408:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828219.074645:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828219.074656:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828219/real 1787828219] req@ffff9e680a2d4b40 x1874668790433408/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828219.074668:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790433408/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828219.074677:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828219.074680:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828219.074684:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790433408:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828219.074690:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790433536:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828219.074727:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828219.074739:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828219/real 1787828219] req@ffff9e680a2d61c0 x1874668790433536/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828219.074752:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790433536/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828219.074761:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828219.074764:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828219.074768:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790433536:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828219.585792:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828219.777848:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828220.609781:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828220.801890:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828221.633786:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828221.825883:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828222.657819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828222.849864:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828223.681762:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828223.873860:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828224.193783:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828224.193791:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828224.193794:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828224.193802:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4860 00000100:00080000:0.0:1787828224.193812:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828224.193818:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828224.193869:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790433664/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828224.193896:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828224.193899:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828224.193901:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828224.193905:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4860 00000100:00080000:0.0:1787828224.193908:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828224.193911:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828224.193919:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828224.193945:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790433792/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828224.193953:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828224.193955:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828224.193957:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828224.193959:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4860 00000100:00080000:0.0:1787828224.193962:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828224.193965:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828224.193968:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828224.193975:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790433920/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828224.193982:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828224.193985:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828224.193986:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828224.193989:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4860 00000100:00080000:0.0:1787828224.194010:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828224.194011:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790433664:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828224.194023:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828224.194032:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828224.194065:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790434048/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828224.194080:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828224.194085:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828224.194088:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828224.194095:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4860 00000100:00080000:0.0:1787828224.194101:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828224.194108:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828224.194114:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828224.194122:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828224.194132:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790434176/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828224.194147:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828224.194151:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828224.194154:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828224.194160:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4860 00000100:00080000:0.0:1787828224.194167:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828224.194173:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828224.194184:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828224.194188:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828224/real 1787828224] req@ffff9e6803e992c0 x1874668790433664/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828224.194207:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790434304/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828224.194213:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790433664/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828224.194255:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828224.194259:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828224.194261:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828224.194269:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828224.194275:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787828224.194284:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4860 00000100:00100000:2.0:1787828224.194288:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790433664:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828224.194295:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828224.194302:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790433792:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828224.194310:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828224.194323:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828224.194340:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828224.194343:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790434432/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828224.194355:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828224.194363:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828224/real 1787828224] req@ffff9e6803e9ad00 x1874668790433792/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828224.194368:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828224.194374:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828224.194385:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4860 00000100:00080000:0.0:1787828224.194393:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828224.194396:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790433792/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828224.194409:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787828224.194412:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828224.194421:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:0.0:1787828224.194424:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828224.194425:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790433792:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828224.194461:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790434560/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828224.194463:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790433920:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828224.194491:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828224.194511:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828224.194514:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828224.194524:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4860 00000800:00000100:2.0:1787828224.194525:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828224.194534:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828224.194540:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828224/real 1787828224] req@ffff9e6803e98b40 x1874668790433920/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828224.194542:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828224.194559:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828224.194571:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790433920/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828224.194575:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790434688/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828224.194624:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828224.194649:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828224.194655:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790433920:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828224.194668:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790434048:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828224.194691:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828224.194745:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828224/real 1787828224] req@ffff9e6803e9bc00 x1874668790434048/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828224.194758:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790434048/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828224.194768:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828224.194772:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828224.194776:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790434048:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828224.194783:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790434176:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828224.194805:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828224.194814:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828224/real 1787828224] req@ffff9e6803e98780 x1874668790434176/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828224.194825:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790434176/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828224.194833:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828224.194835:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828224.194838:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790434176:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828224.194842:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790434304:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828224.194858:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828224.194866:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828224/real 1787828224] req@ffff9e6803e9b480 x1874668790434304/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828224.194876:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790434304/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828224.194885:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828224.194887:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828224.194889:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790434304:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828224.194894:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790434432:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828224.194911:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828224.194933:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828224/real 1787828224] req@ffff9e6803e9a1c0 x1874668790434432/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828224.194946:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790434432/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828224.194955:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828224.194959:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828224.194963:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790434432:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828224.194995:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790434560:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828224.195012:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828224.195020:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828224/real 1787828224] req@ffff9e6803e9a580 x1874668790434560/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828224.195028:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790434560/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828224.195039:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828224.195041:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828224.195043:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790434560:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828224.195047:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790434688:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828224.195078:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828224.195094:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828224/real 1787828224] req@ffff9e6803e99680 x1874668790434688/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828224.195103:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790434688/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828224.195108:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828224.195111:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828224.195113:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790434688:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828224.705811:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828224.897930:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828225.729779:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828225.921917:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828226.753859:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828226.945914:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828227.777796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828227.969908:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828228.801785:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828228.993859:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828229.313785:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828229.313796:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828229.313800:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828229.313812:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4866 00000100:00080000:3.0:1787828229.313825:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828229.313833:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828229.313874:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790434816/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828229.313903:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828229.313907:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828229.313909:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828229.313912:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4866 00000100:00080000:3.0:1787828229.313915:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828229.313918:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828229.313928:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828229.313943:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790434944/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828229.313944:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790434816:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828229.313959:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828229.313962:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828229.313963:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828229.313966:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4866 00000100:00080000:3.0:1787828229.313968:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828229.313972:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828229.313976:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828229.313985:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790435072/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828229.313993:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828229.313996:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828229.313999:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828229.314002:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4866 00000100:00080000:3.0:1787828229.314028:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828229.314034:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828229.314039:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828229.314045:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828229.314053:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790435200/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828229.314062:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828229.314065:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828229.314067:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828229.314071:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4866 00000100:00080000:3.0:1787828229.314075:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828229.314078:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828229.314082:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828229.314092:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828229/real 1787828229] req@ffff9e680a2d6580 x1874668790434816/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828229.314095:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790435328/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828229.314125:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828229.314128:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828229.314131:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828229.314136:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4866 00000100:00100000:2.0:1787828229.314137:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790434816/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828229.314140:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828229.314148:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787828229.314154:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828229.314154:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828229.314163:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828229.314168:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790434816:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828229.314170:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790435456/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828229.314188:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828229.314191:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828229.314192:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828229.314195:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4866 00000100:00080000:3.0:1787828229.314198:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828229.314201:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790434944:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828229.314202:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787828229.314226:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828229.314236:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828229/real 1787828229] req@ffff9e680a2d70c0 x1874668790434944/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828229.314244:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790434944/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828229.314251:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828229.314253:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828229.314256:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790434944:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828229.314260:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790435072:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828229.314272:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828229.314281:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828229/real 1787828229] req@ffff9e680a2d4000 x1874668790435072/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828229.314282:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828229.314295:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790435072/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828229.314301:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828229.314303:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828229.314306:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790435072:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828229.314309:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790435200:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828229.314321:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828229.314328:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828229/real 1787828229] req@ffff9e680a2d5a40 x1874668790435200/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828229.314337:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790435200/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828229.314342:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828229.314344:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828229.314346:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790435200:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828229.314350:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790435328:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828229.314361:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828229.314367:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828229/real 1787828229] req@ffff9e680a2d43c0 x1874668790435328/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828229.314376:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790435328/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828229.314381:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828229.314383:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828229.314385:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790435328:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828229.314390:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790435456:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828229.314401:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828229.314409:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828229/real 1787828229] req@ffff9e680a2d61c0 x1874668790435456/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828229.314414:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790435584/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828229.314424:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790435456/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828229.314431:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828229.314441:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828229.314445:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828229.314450:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828229.314451:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4866 00000100:00080000:2.0:1787828229.314456:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828229.314459:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828229.314461:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790435456:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828229.314480:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790435584:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828229.314495:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828229.314509:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828229/real 1787828229] req@ffff9e680a2d4b40 x1874668790435584/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828229.314517:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828229.314526:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790435584/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828229.314530:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828229.314534:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828229.314538:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828229.314540:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790435584:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828229.314545:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790435712/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828229.314557:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828229.314560:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828229.314562:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828229.314565:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4866 00000100:00080000:3.0:1787828229.314570:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828229.314573:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828229.314576:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790435712:192.168.203.109@tcp:8:kworker.0 00000008:00100000:3.0:1787828229.314579:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828229.314593:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790435840/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828229.314600:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828229.314616:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828229/real 1787828229] req@ffff9e680a2d52c0 x1874668790435712/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828229.314627:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790435712/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828229.314633:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828229.314635:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828229.314638:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790435712:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828229.314648:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790435840:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828229.314667:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828229.314678:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828229/real 1787828229] req@ffff9e680a2d4f00 x1874668790435840/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828229.314692:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790435840/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828229.314726:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828229.314729:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828229.314732:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790435840:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828229.825763:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828230.017876:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828230.849764:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828231.041858:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828231.873765:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828232.065972:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828232.897783:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828233.089842:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828233.921796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828234.113952:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828234.433829:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828234.433837:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828234.433840:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828234.433848:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4871 00000100:00080000:0.0:1787828234.433857:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828234.433863:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828234.433907:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790435968/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828234.433929:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828234.433932:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828234.433933:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828234.433936:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4871 00000100:00080000:0.0:1787828234.433939:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828234.433941:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828234.433950:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828234.433959:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790436096/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828234.433965:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790435968:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828234.433991:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828234.433994:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828234.433996:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828234.434000:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4871 00000100:00080000:0.0:1787828234.434002:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828234.434005:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828234.434019:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828234.434028:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790436224/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828234.434035:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828234.434037:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828234.434039:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828234.434041:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4871 00000100:00080000:0.0:1787828234.434044:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828234.434046:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828234.434049:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828234.434056:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790436352/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828234.434062:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828234.434065:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828234.434066:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828234.434069:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4871 00000100:00080000:0.0:1787828234.434073:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828234.434077:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828234.434080:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828234.434089:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790436096:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828234.434090:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790436480/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828234.434106:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828234.434108:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828234.434110:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828234.434112:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4871 00000100:00080000:0.0:1787828234.434115:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828234.434118:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828234.434122:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828234.434130:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790436608/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828234.434138:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790436224:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828234.434139:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828234.434145:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828234.434149:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828234.434152:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4871 00000100:00080000:0.0:1787828234.434155:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828234.434158:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828234.434162:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828234.434170:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790436736/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828234.434178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790436352:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787828234.434196:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828234.434199:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828234.434202:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828234.434208:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4871 00000100:00080000:0.0:1787828234.434213:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828234.434218:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790436480:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787828234.434224:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828234.434232:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828234.434237:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790436608:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828234.434249:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790436864/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828234.434256:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790436736:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828234.434269:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828234.434274:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828234.434275:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828234.434280:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4871 00000100:00080000:0.0:1787828234.434285:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828234.434290:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828234.434301:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828234.434307:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790436864:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828234.434316:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790436992/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828234.434340:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790436992:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828234.945862:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828235.138004:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828235.969839:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828236.162024:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828236.993842:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828237.185854:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787828237.506268:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:3.0:1787828237.506347:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787828237.506390:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787828237.506455:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:3.0:1787828237.506471:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787828237.506481:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787828237.506494:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787828237.506506:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787828237.506517:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787828237.506520:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828234/real 1787828237] req@ffff9e6803e98780 x1874668790436992/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787828237.506536:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787828237.506560:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787828237.506587:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790436992/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828237.506603:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828237.506609:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828237.506614:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790436992:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828237.506630:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828234/real 1787828237] req@ffff9e6803e9b480 x1874668790436864/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828237.506646:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790436864/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828237.506656:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828237.506658:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828237.506661:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790436864:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828237.506667:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828234/real 1787828237] req@ffff9e6803e99680 x1874668790436480/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828237.506676:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790436480/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828237.506681:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828237.506684:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828237.506686:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790436480:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828237.506722:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828234/real 1787828237] req@ffff9e6803e9a580 x1874668790436608/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828237.506741:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790436608/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828237.506751:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828237.506755:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828237.506759:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790436608:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828237.506767:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828234/real 1787828237] req@ffff9e6803e9a1c0 x1874668790436736/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828237.506778:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790436736/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828237.506785:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828237.506789:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828237.506795:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790436736:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828237.506803:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828234/real 1787828237] req@ffff9e6803e98f00 x1874668790436096/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828237.506817:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790436096/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828237.506828:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828237.506832:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828237.506837:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790436096:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828237.506846:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828234/real 1787828237] req@ffff9e6803e9b840 x1874668790436224/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828237.506859:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790436224/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828237.506867:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828237.506871:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828237.506875:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790436224:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828237.506882:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828234/real 1787828237] req@ffff9e6803e9a940 x1874668790436352/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828237.506894:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790436352/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828237.506902:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828237.506906:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828237.506909:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790436352:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828237.506916:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828234/real 1787828237] req@ffff9e6803e99e00 x1874668790435968/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828237.506930:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790435968/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828237.506938:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828237.506942:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828237.506945:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790435968:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828238.017864:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828238.210014:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828239.041837:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828239.233898:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828239.553868:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828239.553886:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828239.553895:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828239.553913:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4876 00000100:00080000:1.0:1787828239.553931:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828239.554019:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828239.554078:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790437120/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828239.554138:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828239.554143:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828239.554146:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828239.554153:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4876 00000100:00080000:1.0:1787828239.554159:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828239.554165:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828239.554181:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828239.554198:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790437248/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828239.554214:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828239.554220:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828239.554222:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828239.554229:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4876 00000100:00080000:1.0:1787828239.554234:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828239.554239:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828239.554246:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828239.554260:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790437376/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828239.554277:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828239.554283:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828239.554288:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828239.554295:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4876 00000100:00080000:1.0:1787828239.554301:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828239.554308:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828239.554318:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828239.554322:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790437120:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828239.554342:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790437504/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828239.554364:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828239.554369:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828239.554373:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828239.554379:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4876 00000100:00080000:1.0:1787828239.554386:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828239.554392:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828239.554401:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828239.554422:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790437632/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828239.554437:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828239.554446:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:2.0:1787828239.554447:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828239.554451:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828239.554462:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4876 00000100:00080000:1.0:1787828239.554470:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828239.554478:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828239.554486:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828239.554503:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790437760/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828239.554513:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828239.554517:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828239.554520:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828239.554528:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828239/real 1787828239] req@ffff9e6831aa8000 x1874668790437120/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828239.554545:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4876 00000100:00080000:1.0:1787828239.554555:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828239.554563:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828239.554570:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790437120/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828239.554575:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828239.554600:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828239.554607:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790437888/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828239.554614:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828239.554627:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790437120:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828239.554635:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828239.554643:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787828239.554648:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790437248:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828239.554649:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828239.554667:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4876 00000100:00080000:1.0:1787828239.554676:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828239.554699:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828239.554707:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828239.554763:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790438016/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828239.554767:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828239.554783:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828239.554788:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828239.554791:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828239.554797:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828239/real 1787828239] req@ffff9e6831aa9a40 x1874668790437248/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828239.554803:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4876 00000100:00080000:1.0:1787828239.554813:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828239.554823:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828239.554829:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790437248/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828239.554834:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828239.554852:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828239.554854:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790438144/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828239.554865:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828239.554885:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790437248:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828239.554898:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790437376:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828239.554931:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828239.554972:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828239/real 1787828239] req@ffff9e6831aab480 x1874668790437376/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828239.554990:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790437376/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828239.555001:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828239.555005:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828239.555010:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790437376:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828239.555048:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790437504:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828239.555080:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828239.555098:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828239/real 1787828239] req@ffff9e6831aabc00 x1874668790437504/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828239.555115:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790437504/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828239.555126:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828239.555132:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828239.555137:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790437504:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828239.555146:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790437632:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828239.555179:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828239.555193:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828239/real 1787828239] req@ffff9e6831aaad00 x1874668790437632/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828239.555209:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790437632/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828239.555228:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828239.555233:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828239.555237:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790437632:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828239.555245:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790437760:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828239.555271:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828239.555286:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828239/real 1787828239] req@ffff9e6831aaa1c0 x1874668790437760/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828239.555302:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790437760/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828239.555315:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828239.555320:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828239.555324:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790437760:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828239.555333:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790437888:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828239.555359:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828239.555375:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828239/real 1787828239] req@ffff9e6831aa8780 x1874668790437888/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828239.555390:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790437888/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828239.555401:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828239.555404:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828239.555409:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790437888:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828239.555415:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790438016:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828239.555457:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828239.555472:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828239/real 1787828239] req@ffff9e6831aaa940 x1874668790438016/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828239.555485:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790438016/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828239.555495:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828239.555498:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828239.555502:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790438016:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828239.555508:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790438144:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828239.555533:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828239.555576:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828239/real 1787828239] req@ffff9e6831aa9e00 x1874668790438144/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828239.555591:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790438144/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828239.555599:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828239.555603:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828239.555607:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790438144:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828240.065781:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828240.257861:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828241.089807:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828241.281856:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828242.113784:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828242.306321:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828243.137827:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828243.330025:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828244.161822:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828244.353861:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828244.673818:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828244.673826:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828244.673830:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828244.673850:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4881 00000100:00080000:0.0:1787828244.673860:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828244.673865:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828244.673904:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790438272/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828244.673947:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828244.673951:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828244.673953:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828244.673956:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4881 00000100:00080000:0.0:1787828244.673959:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828244.673963:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828244.673972:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828244.673983:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790438400/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828244.673992:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828244.673995:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790438272:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828244.673996:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828244.673999:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828244.674008:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4881 00000100:00080000:0.0:1787828244.674011:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828244.674013:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828244.674015:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828244.674024:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790438528/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828244.674031:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828244.674034:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828244.674036:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828244.674039:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4881 00000100:00080000:0.0:1787828244.674042:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828244.674044:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828244.674047:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828244.674055:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790438656/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828244.674061:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828244.674062:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828244.674065:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828244.674068:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828244.674071:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4881 00000100:00080000:0.0:1787828244.674075:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828244.674078:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828244.674081:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828244.674090:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790438784/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828244.674106:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828244.674109:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828244.674110:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828244.674116:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4881 00000100:00100000:2.0:1787828244.674118:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828244/real 1787828244] req@ffff9e6803e9bc00 x1874668790438272/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828244.674119:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828244.674127:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828244.674136:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828244.674141:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790438272/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828244.674150:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828244.674152:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790438912/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828244.674155:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828244.674163:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790438272:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828244.674175:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828244.674179:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828244.674182:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828244.674186:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4881 00000100:00080000:0.0:1787828244.674190:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828244.674190:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790438400:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828244.674198:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828244.674203:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828244.674218:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790439040/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828244.674218:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828244.674229:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828244.674232:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828244/real 1787828244] req@ffff9e6803e98b40 x1874668790438400/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828244.674233:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828244.674237:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828244.674244:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4881 00000100:00080000:0.0:1787828244.674249:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828244.674253:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828244.674257:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790438400/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828244.674258:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828244.674269:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828244.674271:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790439168/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828244.674275:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828244.674283:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790438400:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828244.674289:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828244.674293:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790438528:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828244.674294:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828244.674299:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828244.674303:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4881 00000100:00080000:0.0:1787828244.674306:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828244.674309:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828244.674313:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828244.674315:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828244.674325:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790439296/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828244.674327:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828244/real 1787828244] req@ffff9e6803e9ad00 x1874668790438528/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828244.674352:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790438528/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828244.674362:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828244.674365:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828244.674367:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790438528:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828244.674371:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790438656:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828244.674383:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828244.674388:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828244/real 1787828244] req@ffff9e6803e992c0 x1874668790438656/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828244.674395:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790438656/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828244.674399:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828244.674401:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828244.674403:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790438656:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828244.674406:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790438784:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828244.674415:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828244.674420:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828244/real 1787828244] req@ffff9e6803e99e00 x1874668790438784/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828244.674427:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790438784/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828244.674432:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828244.674434:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828244.674436:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790438784:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828244.674439:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790438912:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828244.674449:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828244.674454:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828244/real 1787828244] req@ffff9e6803e9a940 x1874668790438912/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828244.674460:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790438912/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828244.674466:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828244.674468:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828244.674470:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790438912:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828244.674481:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790439040:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828244.674491:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828244.674496:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828244/real 1787828244] req@ffff9e6803e9b840 x1874668790439040/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828244.674503:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790439040/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828244.674508:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828244.674510:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828244.674511:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790439040:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828244.674515:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790439168:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828244.674524:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828244.674530:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828244/real 1787828244] req@ffff9e6803e98f00 x1874668790439168/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828244.674537:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790439168/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828244.674542:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828244.674544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828244.674545:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790439168:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828244.674548:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790439296:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828244.674557:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828244.674562:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828244/real 1787828244] req@ffff9e6803e9a1c0 x1874668790439296/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828244.674569:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790439296/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828244.674574:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828244.674575:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828244.674577:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790439296:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828245.185794:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828245.377924:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828246.209882:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828246.401951:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828247.233877:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828247.426026:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828248.257856:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828248.449940:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828249.281801:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828249.473858:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828249.793799:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828249.793807:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828249.793811:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828249.793819:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4886 00000100:00080000:1.0:1787828249.793828:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828249.793834:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828249.793867:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790439424/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828249.793917:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828249.793919:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828249.793921:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828249.793924:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4886 00000100:00080000:1.0:1787828249.793927:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828249.793929:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828249.793938:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828249.793948:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790439552/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828249.793956:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828249.793960:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828249.793962:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790439424:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828249.793963:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828249.793970:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4886 00000100:00080000:1.0:1787828249.793973:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828249.793975:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828249.793978:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828249.793986:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790439680/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828249.793992:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828249.793994:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828249.793996:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828249.793998:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4886 00000100:00080000:1.0:1787828249.794001:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828249.794004:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828249.794007:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828249.794014:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790439808/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828249.794020:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828249.794023:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828249.794024:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828249.794029:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4886 00000800:00000100:2.0:1787828249.794030:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828249.794032:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828249.794036:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828249.794039:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828249.794048:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790439936/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828249.794054:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828249.794057:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828249.794058:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828249.794060:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4886 00000100:00080000:1.0:1787828249.794063:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828249.794066:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828249.794070:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828249/real 1787828249] req@ffff9e6831aaa580 x1874668790439424/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828249.794071:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828249.794083:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790440064/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828249.794091:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790439424/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828249.794106:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828249.794111:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828249.794112:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828249.794114:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828249.794116:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4886 00000100:00080000:1.0:1787828249.794120:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828249.794120:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828249.794125:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828249.794129:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790439424:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787828249.794131:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828249.794141:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790440192/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828249.794147:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828249.794149:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828249.794151:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828249.794153:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790439552:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828249.794157:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4886 00000100:00080000:1.0:1787828249.794162:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828249.794165:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828249.794168:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828249.794176:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790440320/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828249.794181:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828249.794192:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828249.794194:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828249.794197:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828249.794197:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828249/real 1787828249] req@ffff9e6831aa9680 x1874668790439552/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828249.794205:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4886 00000100:00080000:1.0:1787828249.794211:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828249.794216:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828249.794219:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828249.794221:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790439552/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828249.794229:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790440448/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828249.794231:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828249.794240:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828249.794246:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790439552:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828249.794251:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790439680:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828249.794265:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828249.794272:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828249/real 1787828249] req@ffff9e6831aa8b40 x1874668790439680/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828249.794279:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790439680/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828249.794284:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828249.794286:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828249.794288:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790439680:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828249.794292:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790439808:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828249.794302:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828249.794308:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828249/real 1787828249] req@ffff9e6831aa8f00 x1874668790439808/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828249.794315:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790439808/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828249.794322:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828249.794325:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828249.794329:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790439808:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828249.794334:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790439936:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828249.794354:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828249.794365:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828249/real 1787828249] req@ffff9e6831aa9e00 x1874668790439936/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828249.794378:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790439936/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828249.794384:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828249.794386:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828249.794388:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790439936:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828249.794392:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790440064:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828249.794407:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828249.794414:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828249/real 1787828249] req@ffff9e6831aaa940 x1874668790440064/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828249.794431:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790440064/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828249.794438:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828249.794440:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828249.794442:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790440064:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828249.794445:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790440192:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828249.794456:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828249.794462:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828249/real 1787828249] req@ffff9e6831aa8780 x1874668790440192/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828249.794469:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790440192/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828249.794474:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828249.794476:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828249.794478:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790440192:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828249.794490:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790440320:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828249.794500:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828249.794506:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828249/real 1787828249] req@ffff9e6831aaa1c0 x1874668790440320/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828249.794513:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790440320/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828249.794523:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828249.794525:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828249.794527:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790440320:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828249.794530:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790440448:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828249.794539:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828249.794544:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828249/real 1787828249] req@ffff9e6831aaad00 x1874668790440448/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828249.794552:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790440448/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828249.794561:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828249.794564:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828249.794567:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790440448:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828250.305941:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828250.497955:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828251.329839:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828251.521913:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828252.353811:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828252.545846:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828253.377921:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828253.570034:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828254.401799:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828254.594004:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828254.914029:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828254.914049:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828254.914058:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828254.914095:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4891 00000100:00080000:3.0:1787828254.914115:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828254.914128:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828254.914197:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790440576/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828254.914260:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828254.914266:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828254.914270:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828254.914277:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4891 00000100:00080000:3.0:1787828254.914284:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828254.914291:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828254.914308:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828254.914332:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790440704/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828254.914354:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828254.914356:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790440576:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828254.914363:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828254.914368:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828254.914376:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4891 00000100:00080000:3.0:1787828254.914386:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828254.914396:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828254.914405:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828254.914428:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790440832/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828254.914445:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828254.914451:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828254.914455:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828254.914462:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4891 00000100:00080000:3.0:1787828254.914484:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828254.914493:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787828254.914516:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828254.914598:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828254/real 1787828254] req@ffff9e680a2d6d00 x1874668790440576/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828254.914635:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790440576/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828254.914655:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828254.914663:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828254.914671:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790440576:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787828254.914797:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790440704:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787828254.914822:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828254.914883:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828254.914919:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828254/real 1787828254] req@ffff9e680a2d7c00 x1874668790440704/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828254.914929:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790440960/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828254.914966:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828254.914967:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790440704/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828254.914980:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828254.914988:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828254.915005:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828254.915006:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4891 00000100:00080000:2.0:1787828254.915022:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828254.915023:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828254.915036:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790440704:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828254.915039:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828254.915059:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828254.915065:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790440832:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828254.915087:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790441088/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828254.915108:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828254.915114:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:2.0:1787828254.915121:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828254.915122:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828254.915136:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4891 00000100:00080000:3.0:1787828254.915146:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828254.915154:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828254.915161:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828254/real 1787828254] req@ffff9e680a2d7480 x1874668790440832/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828254.915168:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828254.915204:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790441216/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828254.915208:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790440832/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828254.915251:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828254.915254:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828254.915263:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828254.915270:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828254.915272:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828254.915284:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790440832:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828254.915287:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4891 00000100:00080000:3.0:1787828254.915299:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828254.915310:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828254.915320:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828254.915330:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790440960:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828254.915345:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790441344/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828254.915368:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828254.915376:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1787828254.915382:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828254.915384:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828254.915398:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4891 00000100:00080000:3.0:1787828254.915407:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828254.915415:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828254.915444:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828254.915447:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828254/real 1787828254] req@ffff9e680a2d5e00 x1874668790440960/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828254.915473:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790441472/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828254.915483:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790440960/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828254.915518:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828254.915533:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828254.915534:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828254.915547:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790440960:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787828254.915548:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828254.915556:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828254.915570:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790441088:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787828254.915571:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4891 00000100:00080000:3.0:1787828254.915584:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828254.915595:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828254.915622:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828254.915644:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828254.915650:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790441600/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828254.915678:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828254/real 1787828254] req@ffff9e680a2d4780 x1874668790441088/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828254.915721:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790441088/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828254.915737:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828254.915744:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828254.915751:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790441088:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828254.915816:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790441216:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828254.915884:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828254.915905:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828254/real 1787828254] req@ffff9e680a2d6d00 x1874668790441216/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828254.915926:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790441216/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828254.915954:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828254.915960:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828254.915967:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790441216:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828254.916010:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790441344:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828254.916047:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828254.916080:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828254/real 1787828254] req@ffff9e680a2d4f00 x1874668790441344/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828254.916101:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790441344/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828254.916115:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828254.916122:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828254.916127:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790441344:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828254.916136:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790441472:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828254.916164:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828254.916183:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828254/real 1787828254] req@ffff9e680a2d52c0 x1874668790441472/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828254.916204:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790441472/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828254.916219:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828254.916225:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828254.916232:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790441472:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828254.916243:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790441600:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828254.916278:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828254.916294:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828254/real 1787828254] req@ffff9e680a2d4b40 x1874668790441600/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828254.916307:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790441600/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828254.916317:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828254.916320:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828254.916323:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790441600:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828255.425807:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828255.617952:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828256.449885:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828256.641920:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828257.473946:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828257.666132:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828258.497788:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828258.689916:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828259.521769:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828259.713907:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828260.033798:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828260.033818:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828260.033822:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828260.033830:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4896 00000100:00080000:1.0:1787828260.033839:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828260.033844:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828260.033878:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790441728/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828260.033899:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828260.033902:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828260.033903:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828260.033906:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4896 00000100:00080000:1.0:1787828260.033908:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828260.033911:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828260.033919:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828260.033928:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790441856/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828260.033940:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828260.033942:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828260.033943:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828260.033947:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4896 00000100:00080000:1.0:1787828260.033950:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828260.033952:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828260.033956:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828260.033963:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790441984/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828260.033972:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790441728:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828260.033977:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828260.033988:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828260.033989:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828260.033993:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4896 00000100:00080000:1.0:1787828260.033996:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828260.033998:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828260.034001:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828260.034011:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790442112/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828260.034016:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828260.034025:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828260.034027:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828260.034029:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4896 00000100:00080000:1.0:1787828260.034034:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828260.034036:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828260.034041:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828260.034048:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790442240/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828260.034055:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828260.034058:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828260.034059:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828260.034061:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4896 00000100:00080000:1.0:1787828260.034064:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828260.034067:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828260.034071:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828260.034079:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828260.034085:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790442368/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828260.034094:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828260.034096:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828260.034099:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828260.034104:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4896 00000100:00080000:1.0:1787828260.034113:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828260.034115:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828260.034119:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828260.034126:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790442496/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828260.034137:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828260.034143:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828260/real 1787828260] req@ffff9e6831aabc00 x1874668790441728/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828260.034144:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828260.034157:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828260.034170:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4896 00000100:00080000:1.0:1787828260.034176:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828260.034180:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790441728/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828260.034189:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828260.034199:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828260.034216:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790442624/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828260.034217:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828260.034232:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828260.034234:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828260.034240:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828260.034246:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828260.034246:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790441728:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828260.034263:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4896 00000100:00080000:1.0:1787828260.034267:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828260.034272:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790441856:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828260.034273:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828260.034288:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828260.034302:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790442752/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828260.034321:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828260.034339:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828260/real 1787828260] req@ffff9e6831aab480 x1874668790441856/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828260.034352:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790441856/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828260.034363:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828260.034367:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828260.034371:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790441856:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828260.034402:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790441984:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828260.034422:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828260.034433:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828260/real 1787828260] req@ffff9e6831aa9a40 x1874668790441984/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828260.034446:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790441984/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828260.034455:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828260.034458:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828260.034462:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790441984:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828260.034468:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790442112:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828260.034484:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828260.034493:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828260/real 1787828260] req@ffff9e6831aa8000 x1874668790442112/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828260.034505:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790442112/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828260.034515:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828260.034519:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828260.034522:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790442112:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828260.034528:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790442240:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828260.034543:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828260.034552:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828260/real 1787828260] req@ffff9e6831aaad00 x1874668790442240/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828260.034563:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790442240/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828260.034572:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828260.034576:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828260.034579:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790442240:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828260.034585:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790442368:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828260.034601:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828260.034610:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828260/real 1787828260] req@ffff9e6831aaa1c0 x1874668790442368/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828260.034623:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790442368/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828260.034633:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828260.034637:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828260.034641:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790442368:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828260.034648:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790442496:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828260.034666:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828260.034677:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828260/real 1787828260] req@ffff9e6831aa8780 x1874668790442496/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828260.034689:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790442496/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828260.034726:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828260.034730:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828260.034734:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790442496:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828260.034741:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790442624:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828260.034759:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828260.034769:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828260/real 1787828260] req@ffff9e6831aaa940 x1874668790442624/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828260.034782:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790442624/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828260.034791:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828260.034794:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828260.034798:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790442624:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828260.034817:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790442752:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828260.034836:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828260.034847:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828260/real 1787828260] req@ffff9e6831aa9e00 x1874668790442752/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828260.034860:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790442752/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828260.034868:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828260.034872:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828260.034875:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790442752:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828260.545873:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828260.737982:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828261.569855:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828261.761930:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828262.593779:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828262.785941:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828263.617892:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828263.810000:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828264.641902:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828264.834012:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828265.153887:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828265.153909:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828265.153920:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828265.153942:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4901 00000100:00080000:2.0:1787828265.153977:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828265.153993:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828265.154073:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790442880/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828265.154140:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828265.154146:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828265.154149:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828265.154158:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4901 00000100:00080000:2.0:1787828265.154171:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828265.154183:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828265.154200:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828265.154227:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790443008/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828265.154240:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790442880:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828265.154257:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828265.154265:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828265.154269:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828265.154278:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4901 00000100:00080000:2.0:1787828265.154286:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828265.154295:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828265.154304:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828265.154329:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790443136/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828265.154345:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828265.154354:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787828265.154356:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828265.154360:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828265.154369:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4901 00000100:00080000:2.0:1787828265.154375:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828265.154381:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828265.154392:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828265.154417:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790443264/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828265.154431:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828265/real 1787828265] req@ffff9e6804423840 x1874668790442880/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828265.154445:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828265.154473:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828265.154486:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828265.154494:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4901 00000100:00080000:2.0:1787828265.154505:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828265.154514:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828265.154522:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790442880/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828265.154523:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828265.154556:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787828265.154557:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790443392/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828265.154588:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787828265.154611:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828265.154623:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790442880:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828265.154625:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828265.154635:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828265.154656:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4901 00000100:00080000:2.0:1787828265.154664:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828265.154673:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828265.154681:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828265.154748:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790443008:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828265.154749:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790443520/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828265.154823:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828265.154831:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828265.154835:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828265.154850:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4901 00000100:00080000:2.0:1787828265.154866:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828265.154867:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828265.154891:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828265.154906:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828265/real 1787828265] req@ffff9e68044221c0 x1874668790443008/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828265.154914:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828265.154938:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790443008/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828265.154958:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787828265.154963:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790443648/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828265.154972:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828265.154997:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790443008:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828265.155036:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790443136:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828265.155075:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828265.155091:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787828265.155093:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828265.155109:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828265.155122:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828265/real 1787828265] req@ffff9e68044230c0 x1874668790443136/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828265.155123:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4901 00000100:00080000:2.0:1787828265.155138:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828265.155157:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828265.155162:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790443136/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828265.155170:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828265.155206:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787828265.155215:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790443776/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828265.155220:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828265.155249:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790443136:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828265.155263:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828265.155275:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790443264:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787828265.155276:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828265.155288:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828265.155302:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4901 00000100:00080000:2.0:1787828265.155311:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828265.155317:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828265.155325:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828265.155329:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828265.155356:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790443904/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828265.155359:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828265/real 1787828265] req@ffff9e68044212c0 x1874668790443264/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828265.155456:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790443264/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828265.155471:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828265.155476:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828265.155483:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790443264:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828265.155501:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790443392:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828265.155523:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828265.155535:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828265/real 1787828265] req@ffff9e6804423480 x1874668790443392/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828265.155548:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790443392/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828265.155558:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828265.155561:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828265.155566:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790443392:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828265.155589:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790443520:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828265.155610:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828265.155622:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828265/real 1787828265] req@ffff9e6804420b40 x1874668790443520/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828265.155636:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790443520/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828265.155648:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828265.155652:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828265.155655:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790443520:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828265.155662:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790443648:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828265.155681:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828265.155691:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828265/real 1787828265] req@ffff9e6804422580 x1874668790443648/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828265.155749:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790443648/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828265.155761:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828265.155786:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828265.155791:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790443648:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828265.155804:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790443776:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828265.155834:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828265.155852:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828265/real 1787828265] req@ffff9e6804421680 x1874668790443776/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828265.155876:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790443776/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828265.155888:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828265.155893:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828265.155898:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790443776:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828265.155907:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790443904:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828265.155927:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828265.155938:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828265/real 1787828265] req@ffff9e6804420780 x1874668790443904/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828265.155951:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790443904/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828265.155961:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828265.155964:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828265.155968:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790443904:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828265.665848:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828265.857880:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828266.689849:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828266.882007:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828267.713874:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828267.906067:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828268.738149:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828268.930025:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828269.761893:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828269.953954:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828270.273879:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828270.273892:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828270.273898:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828270.273911:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4906 00000100:00080000:2.0:1787828270.273926:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828270.273934:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828270.274006:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790444032/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828270.274057:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828270.274061:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828270.274062:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828270.274066:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4906 00000100:00080000:2.0:1787828270.274069:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828270.274071:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828270.274097:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828270.274108:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790444160/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828270.274117:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828270.274121:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828270.274122:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828270.274128:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4906 00000100:00080000:2.0:1787828270.274131:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828270.274134:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828270.274139:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828270.274149:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790444288/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828270.274159:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828270.274170:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787828270.274171:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790444032:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828270.274173:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828270.274189:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4906 00000100:00080000:2.0:1787828270.274192:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828270.274196:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828270.274201:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828270.274215:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790444416/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828270.274222:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828270.274225:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828270.274227:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828270.274230:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4906 00000100:00080000:2.0:1787828270.274233:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828270.274235:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828270.274241:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828270.274251:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790444544/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828270.274259:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828270.274263:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828270.274264:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828270.274272:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4906 00000800:00000100:1.0:1787828270.274277:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828270.274285:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828270.274293:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828270.274301:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828270.274310:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790444672/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828270.274317:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828270.274320:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828270.274322:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828270.274328:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4906 00000100:00080000:2.0:1787828270.274331:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828270.274333:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828270.274336:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828270.274346:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790444800/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828270.274356:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828270.274364:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828270/real 1787828270] req@ffff9e6804422d00 x1874668790444032/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828270.274364:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828270.274366:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828270.274383:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4906 00000100:00080000:2.0:1787828270.274387:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828270.274398:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828270.274414:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828270.274438:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790444928/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828270.274458:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828270.274466:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828270.274469:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828270.274476:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790444032/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828270.274485:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4906 00000100:00080000:2.0:1787828270.274504:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828270.274510:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828270.274512:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787828270.274532:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:2.0:1787828270.274533:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828270.274543:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790444032:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787828270.274555:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790445056/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828270.274562:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790444160:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828270.274629:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828270.274653:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828270/real 1787828270] req@ffff9e6804420f00 x1874668790444160/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828270.274669:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790444160/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828270.274682:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828270.274686:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828270.274690:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790444160:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828270.274817:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790444288:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828270.274851:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828270.274866:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828270/real 1787828270] req@ffff9e6804421e00 x1874668790444288/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828270.274882:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790444288/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828270.274901:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828270.274905:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828270.274909:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790444288:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828270.274917:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790444416:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828270.274938:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828270.274969:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828270/real 1787828270] req@ffff9e6804422940 x1874668790444416/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828270.274985:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790444416/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828270.274995:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828270.274999:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828270.275003:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790444416:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828270.275010:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790444544:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828270.275030:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828270.275041:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828270/real 1787828270] req@ffff9e6804420780 x1874668790444544/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828270.275055:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790444544/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828270.275065:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828270.275068:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828270.275073:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790444544:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828270.275081:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790444672:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828270.275099:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828270.275110:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828270/real 1787828270] req@ffff9e6804421680 x1874668790444672/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828270.275124:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790444672/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828270.275135:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828270.275139:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828270.275142:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790444672:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828270.275149:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790444800:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828270.275167:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828270.275178:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828270/real 1787828270] req@ffff9e6804422580 x1874668790444800/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828270.275191:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790444800/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828270.275201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828270.275204:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828270.275208:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790444800:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828270.275214:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790444928:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828270.275233:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828270.275245:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828270/real 1787828270] req@ffff9e6804420b40 x1874668790444928/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828270.275259:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790444928/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828270.275268:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828270.275272:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828270.275275:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790444928:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828270.275282:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790445056:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828270.275300:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828270.275311:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828270/real 1787828270] req@ffff9e6804423480 x1874668790445056/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828270.275323:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790445056/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828270.275333:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828270.275337:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828270.275340:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790445056:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828270.785867:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828270.978076:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828271.809817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828272.001980:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828272.833827:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828273.025867:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828273.857865:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828274.049924:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828274.881958:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828275.073918:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828275.393873:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828275.393886:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828275.393891:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828275.393904:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4912 00000100:00080000:2.0:1787828275.393917:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828275.393926:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828275.393973:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790445184/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828275.394010:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828275.394015:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828275.394017:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828275.394022:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4912 00000100:00080000:2.0:1787828275.394026:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828275.394031:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828275.394042:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828275.394058:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790445312/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828275.394082:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828275.394087:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828275.394090:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828275.394095:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4912 00000100:00080000:2.0:1787828275.394098:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828275.394103:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828275.394110:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790445184:192.168.203.109@tcp:250:kworker.0 00000008:00100000:2.0:1787828275.394112:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828275.394133:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790445440/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828275.394147:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828275.394151:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828275.394153:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828275.394157:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4912 00000100:00080000:2.0:1787828275.394161:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828275.394165:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828275.394171:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828275.394184:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790445568/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828275.394196:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828275.394200:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828275.394202:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828275.394206:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4912 00000800:00000100:1.0:1787828275.394215:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828275.394216:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828275.394226:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828275.394233:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828275.394250:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790445696/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828275.394262:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828275.394266:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828275.394269:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828275.394279:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828275/real 1787828275] req@ffff9e68044212c0 x1874668790445184/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828275.394280:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4912 00000100:00080000:2.0:1787828275.394291:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828275.394300:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828275.394306:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828275.394325:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790445824/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828275.394335:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790445184/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828275.394343:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828275.394359:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828275.394365:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828275.394373:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828275.394375:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4912 00000100:00080000:2.0:1787828275.394385:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828275.394387:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787828275.394395:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828275.394401:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790445184:192.168.203.109@tcp:250:kworker.0 00000008:00100000:2.0:1787828275.394405:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828275.394442:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790445952/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828275.394448:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790445312:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828275.394493:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828275.394501:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828275.394506:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828275.394515:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828275.394524:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4912 00000100:00080000:2.0:1787828275.394537:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828275.394546:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828275.394551:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828275/real 1787828275] req@ffff9e68044230c0 x1874668790445312/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828275.394558:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828275.394583:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790445312/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828275.394589:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790446080/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828275.394625:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828275.394630:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828275.394637:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787828275.394642:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787828275.394649:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790445312:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828275.394650:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828275.394666:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4912 00000100:00100000:1.0:1787828275.394672:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790445440:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828275.394675:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828275.394686:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828275.394748:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828275.394770:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790446208/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828275.394800:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828275.394826:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828275/real 1787828275] req@ffff9e68044221c0 x1874668790445440/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828275.394843:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790445440/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828275.394854:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828275.394859:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828275.394863:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790445440:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828275.394872:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790445568:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828275.394906:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828275.394927:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828275/real 1787828275] req@ffff9e6804423840 x1874668790445568/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828275.394952:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790445568/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828275.394981:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828275.394988:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828275.394995:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790445568:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828275.395009:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790445696:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828275.395048:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828275.395072:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828275/real 1787828275] req@ffff9e6804423480 x1874668790445696/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828275.395097:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790445696/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828275.395109:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828275.395112:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828275.395117:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790445696:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828275.395125:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790445824:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828275.395153:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828275.395167:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828275/real 1787828275] req@ffff9e6804420b40 x1874668790445824/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828275.395181:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790445824/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828275.395193:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828275.395196:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828275.395200:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790445824:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828275.395226:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790445952:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828275.395246:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828275.395258:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828275/real 1787828275] req@ffff9e6804422580 x1874668790445952/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828275.395272:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790445952/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828275.395282:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828275.395285:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828275.395289:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790445952:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828275.395296:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790446080:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828275.395313:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828275.395324:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828275/real 1787828275] req@ffff9e6804421680 x1874668790446080/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828275.395337:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790446080/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828275.395355:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828275.395359:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828275.395363:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790446080:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828275.395370:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790446208:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828275.395387:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828275.395398:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828275/real 1787828275] req@ffff9e6804420780 x1874668790446208/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828275.395410:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790446208/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828275.395423:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828275.395429:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828275.395436:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790446208:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828275.905844:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828276.097967:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828276.929782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828277.121843:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828277.953790:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828278.145822:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828278.977794:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828279.169857:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828280.001791:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828280.193861:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828280.513797:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828280.513806:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828280.513809:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828280.513817:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4917 00000100:00080000:2.0:1787828280.513827:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828280.513833:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828280.513864:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790446336/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828280.513889:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828280.513892:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828280.513893:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828280.513896:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4917 00000100:00080000:2.0:1787828280.513898:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828280.513900:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828280.513908:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828280.513916:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790446464/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828280.513923:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828280.513925:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828280.513926:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828280.513928:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4917 00000100:00080000:2.0:1787828280.513930:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828280.513932:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828280.513934:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828280.513939:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790446592/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828280.513945:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828280.513948:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828280.513950:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828280.513953:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4917 00000100:00080000:2.0:1787828280.513956:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828280.513958:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828280.513961:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828280.513967:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790446720/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828280.513976:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790446336:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828280.513977:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828280.513985:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828280.513987:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828280.513991:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4917 00000100:00080000:2.0:1787828280.513994:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828280.513998:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828280.514004:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828280.514020:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790446848/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828280.514031:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787828280.514037:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828280.514037:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828280.514040:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828280.514045:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4917 00000100:00080000:2.0:1787828280.514049:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828280.514053:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828280.514059:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828280.514069:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790446976/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828280.514076:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828280.514077:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828280/real 1787828280] req@ffff9e6804422940 x1874668790446336/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828280.514081:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828280.514085:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828280.514092:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4917 00000100:00080000:2.0:1787828280.514097:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828280.514100:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828280.514104:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828280.514105:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790446336/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828280.514113:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790447104/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828280.514116:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828280.514126:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787828280.514128:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828280.514133:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787828280.514134:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790446336:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828280.514135:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828280.514145:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4917 00000100:00100000:1.0:1787828280.514146:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790446464:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828280.514153:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828280.514160:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828280.514166:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828280.514168:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828280.514180:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828280/real 1787828280] req@ffff9e6804421e00 x1874668790446464/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828280.514185:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790447232/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828280.514195:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790446464/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828280.514198:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828280.514205:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828280.514208:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828280.514215:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828280.514216:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4917 00000100:00080000:2.0:1787828280.514223:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828280.514224:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787828280.514230:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828280.514231:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790446464:192.168.203.109@tcp:38:kworker.0 00000008:00100000:2.0:1787828280.514238:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828280.514240:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790446592:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828280.514251:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790447360/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828280.514260:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828280.514271:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828280/real 1787828280] req@ffff9e6804420f00 x1874668790446592/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828280.514278:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790446592/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828280.514284:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828280.514287:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828280.514289:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790446592:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828280.514308:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790446720:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828280.514319:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828280.514325:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828280/real 1787828280] req@ffff9e6804422d00 x1874668790446720/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828280.514332:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790446720/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828280.514337:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828280.514339:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828280.514341:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790446720:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828280.514344:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790446848:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828280.514353:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828280.514358:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828280/real 1787828280] req@ffff9e6804420780 x1874668790446848/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828280.514365:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790446848/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828280.514370:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828280.514372:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828280.514374:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790446848:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828280.514377:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790446976:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828280.514386:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828280.514391:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828280/real 1787828280] req@ffff9e6804421680 x1874668790446976/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828280.514398:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790446976/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828280.514405:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828280.514407:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828280.514409:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790446976:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828280.514412:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790447104:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828280.514421:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828280.514426:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828280/real 1787828280] req@ffff9e6804422580 x1874668790447104/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828280.514433:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790447104/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828280.514437:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828280.514439:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828280.514441:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790447104:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828280.514445:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790447232:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828280.514454:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828280.514459:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828280/real 1787828280] req@ffff9e6804420b40 x1874668790447232/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828280.514465:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790447232/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828280.514470:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828280.514472:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828280.514474:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790447232:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828280.514478:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790447360:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828280.514488:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828280.514493:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828280/real 1787828280] req@ffff9e6804423480 x1874668790447360/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828280.514499:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790447360/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828280.514504:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828280.514506:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828280.514508:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790447360:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828281.025822:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828281.217876:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828282.049772:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828282.241828:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828283.073801:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828283.265850:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828284.097845:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828284.289815:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828285.121774:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828285.313836:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828285.633802:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828285.633812:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828285.633816:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828285.633825:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4922 00000100:00080000:1.0:1787828285.633836:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828285.633844:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828285.633880:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790447488/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828285.633907:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828285.633911:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828285.633914:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828285.633918:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4922 00000100:00080000:1.0:1787828285.633923:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828285.633928:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828285.633939:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828285.633945:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790447488:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828285.633957:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790447616/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828285.633969:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828285.633973:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828285.633975:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828285.633979:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4922 00000100:00080000:1.0:1787828285.633983:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828285.633987:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828285.633992:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828285.634002:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828285.634008:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790447744/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828285.634021:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828285.634024:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828285.634026:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828285.634030:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4922 00000100:00080000:1.0:1787828285.634035:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828285.634039:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828285.634046:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828285.634047:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828285/real 1787828285] req@ffff9e6831aa8f00 x1874668790447488/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828285.634064:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790447872/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828285.634066:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790447488/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828285.634077:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828285.634080:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828285.634086:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790447488:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828285.634105:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828285.634108:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828285.634110:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790447616:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828285.634112:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828285.634121:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4922 00000100:00080000:1.0:1787828285.634127:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828285.634132:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787828285.634135:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1787828285.634142:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828285.634148:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828285/real 1787828285] req@ffff9e6831aa8b40 x1874668790447616/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828285.634165:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790447616/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828285.634166:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790448000/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828285.634202:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828285.634202:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828285.634207:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828285.634212:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787828285.634213:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790447616:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828285.634217:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828285.634221:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790447744:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828285.634226:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4922 00000100:00080000:1.0:1787828285.634233:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828285.634238:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828285.634242:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828285.634248:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828285/real 1787828285] req@ffff9e6831aa9680 x1874668790447744/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828285.634250:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828285.634263:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790447744/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828285.634270:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828285.634271:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790448128/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828285.634274:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828285.634280:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790447744:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828285.634290:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790447872:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787828285.634296:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828285.634301:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828285.634304:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828285.634308:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828285.634314:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4922 00000100:00100000:2.0:1787828285.634318:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828285/real 1787828285] req@ffff9e6831aaa580 x1874668790447872/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828285.634323:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828285.634330:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790447872/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828285.634334:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787828285.634343:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787828285.634347:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828285.634347:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828285.634353:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790447872:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828285.634369:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790448256/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828285.634369:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790448000:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787828285.634385:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828285.634387:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828285.634393:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828285.634397:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828285.634399:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828285/real 1787828285] req@ffff9e6831aa8f00 x1874668790448000/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828285.634405:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4922 00000100:00100000:2.0:1787828285.634412:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790448000/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828285.634413:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828285.634422:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828285.634424:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787828285.634429:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828285.634433:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790448000:192.168.203.109@tcp:8:kworker.0 00000008:00100000:1.0:1787828285.634434:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828285.634438:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790448128:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828285.634454:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790448384/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828285.634464:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828285.634471:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828285.634477:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828285/real 1787828285] req@ffff9e6831aa9e00 x1874668790448128/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828285.634478:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828285.634483:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828285.634494:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4922 00000100:00100000:2.0:1787828285.634495:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790448128/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828285.634501:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828285.634505:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828285.634511:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787828285.634511:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828285.634517:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790448128:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787828285.634522:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828285.634528:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790448256:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828285.634539:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790448512/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828285.634545:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828285.634556:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828285/real 1787828285] req@ffff9e6831aaa940 x1874668790448256/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828285.634580:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790448256/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828285.634587:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828285.634589:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828285.634591:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790448256:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828285.634596:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790448384:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828285.634607:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828285.634613:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828285/real 1787828285] req@ffff9e6831aa8780 x1874668790448384/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828285.634620:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790448384/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828285.634645:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828285.634647:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828285.634649:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790448384:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828285.634657:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790448512:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828285.634673:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828285.634680:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828285/real 1787828285] req@ffff9e6831aaa1c0 x1874668790448512/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828285.634687:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790448512/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828285.634710:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828285.634713:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828285.634715:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790448512:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828286.145778:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828286.337848:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828287.169796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828287.361864:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828288.193755:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828288.385865:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828289.217805:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828289.409849:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828290.241818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828290.433862:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828290.753797:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828290.753807:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828290.753812:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828290.753823:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4927 00000100:00080000:2.0:1787828290.753835:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828290.753843:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828290.753886:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790448640/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828290.753913:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828290.753916:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828290.753918:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828290.753922:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4927 00000100:00080000:2.0:1787828290.753925:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828290.753928:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828290.753937:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828290.753950:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790448640:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787828290.753950:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790448768/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828290.753964:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828290.753967:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828290.753969:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828290.753972:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4927 00000100:00080000:2.0:1787828290.753973:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828290.753976:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828290.753979:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828290.753986:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790448896/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828290.753992:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828290.753994:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828290.753995:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828290.753998:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4927 00000100:00080000:2.0:1787828290.753999:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828290.754002:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828290.754005:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828290.754012:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790449024/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828290.754017:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828290.754020:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828290.754022:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828290.754024:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4927 00000100:00080000:2.0:1787828290.754026:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828290.754028:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828290.754031:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828290.754037:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828290.754043:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790449152/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828290.754051:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828290.754054:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828290.754055:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828290.754058:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4927 00000100:00080000:2.0:1787828290.754060:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828290.754063:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828290.754066:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828290.754073:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790449280/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828290.754078:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828290.754080:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828290.754081:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828290.754084:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4927 00000100:00080000:2.0:1787828290.754087:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828290.754090:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828290.754093:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828290.754098:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828290/real 1787828290] req@ffff9e6804423840 x1874668790448640/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828290.754104:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790449408/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828290.754118:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828290.754121:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828290.754123:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828290.754127:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790448640/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828290.754128:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4927 00000100:00080000:2.0:1787828290.754134:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828290.754140:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828290.754145:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828290.754147:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828290.754153:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828290.754154:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790449536/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828290.754159:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790448640:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828290.754171:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828290.754174:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828290.754175:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828290.754179:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4927 00000100:00080000:2.0:1787828290.754181:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828290.754183:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828290.754187:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828290.754191:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790448768:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828290.754196:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790449664/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828290.754219:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828290.754231:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828290/real 1787828290] req@ffff9e68044221c0 x1874668790448768/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828290.754240:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790448768/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828290.754247:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828290.754249:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828290.754252:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790448768:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828290.754258:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790448896:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828290.754271:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828290.754277:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828290/real 1787828290] req@ffff9e68044230c0 x1874668790448896/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828290.754285:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790448896/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828290.754290:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828290.754292:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828290.754294:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790448896:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828290.754298:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790449024:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828290.754316:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828290.754323:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828290/real 1787828290] req@ffff9e68044212c0 x1874668790449024/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828290.754331:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790449024/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828290.754337:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828290.754339:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828290.754341:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790449024:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828290.754345:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790449152:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828290.754355:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828290.754361:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828290/real 1787828290] req@ffff9e6804423480 x1874668790449152/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828290.754369:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790449152/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828290.754375:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828290.754376:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828290.754378:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790449152:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828290.754381:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790449280:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828290.754391:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828290.754397:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828290/real 1787828290] req@ffff9e6804420b40 x1874668790449280/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828290.754404:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790449280/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828290.754410:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828290.754412:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828290.754414:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790449280:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828290.754417:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790449408:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828290.754426:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828290.754432:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828290/real 1787828290] req@ffff9e6804422580 x1874668790449408/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828290.754439:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790449408/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828290.754449:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828290.754451:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828290.754453:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790449408:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828290.754457:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790449536:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828290.754466:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828290.754472:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828290/real 1787828290] req@ffff9e6804421680 x1874668790449536/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828290.754478:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790449536/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828290.754483:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828290.754485:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828290.754487:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790449536:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828290.754500:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790449664:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828290.754511:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828290.754516:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828290/real 1787828290] req@ffff9e6804420780 x1874668790449664/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828290.754523:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790449664/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828290.754528:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828290.754530:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828290.754533:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790449664:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828291.265826:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828291.457872:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828292.289798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828292.481927:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828293.313845:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828293.505865:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828294.337782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828294.529904:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828295.361780:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828295.553903:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828295.873827:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828295.873841:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828295.873847:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828295.873860:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4932 00000100:00080000:2.0:1787828295.873875:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828295.873884:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828295.873935:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790449792/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828295.873966:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828295.873971:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828295.873974:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828295.873979:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4932 00000100:00080000:2.0:1787828295.873985:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828295.874000:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828295.874014:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828295.874023:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790449792:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787828295.874037:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790449920/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828295.874053:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828295.874057:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828295.874058:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828295.874063:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4932 00000100:00080000:2.0:1787828295.874068:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828295.874073:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828295.874079:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828295.874091:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790450048/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828295.874103:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828295.874118:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828295.874122:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828295.874125:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828295.874131:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4932 00000100:00080000:2.0:1787828295.874136:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828295.874144:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828295.874152:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828295/real 1787828295] req@ffff9e6804422d00 x1874668790449792/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828295.874153:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828295.874181:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790449792/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828295.874181:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790450176/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828295.874228:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828295.874230:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828295.874235:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787828295.874235:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828295.874243:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828295.874249:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790449792:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828295.874255:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4932 00000100:00080000:2.0:1787828295.874272:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828295.874281:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828295.874287:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828295.874298:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790449920:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828295.874313:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790450304/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828295.874329:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787828295.874336:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828295.874342:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828295.874346:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828295.874352:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4932 00000100:00100000:1.0:1787828295.874356:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828295/real 1787828295] req@ffff9e6804420f00 x1874668790449920/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828295.874358:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828295.874369:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828295.874384:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828295.874390:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790449920/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828295.874401:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787828295.874402:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790450432/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828295.874418:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828295.874434:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790449920:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828295.874434:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828295.874451:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828295.874454:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828295.874455:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790450048:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828295.874463:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4932 00000100:00080000:2.0:1787828295.874468:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828295.874471:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828295.874477:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828295.874483:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828295.874490:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790450560/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828295.874499:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828295/real 1787828295] req@ffff9e6804421e00 x1874668790450048/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828295.874512:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828295.874523:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828295.874528:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828295.874529:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790450048/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828295.874538:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4932 00000100:00080000:2.0:1787828295.874546:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828295.874571:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828295.874574:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787828295.874579:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:2.0:1787828295.874582:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828295.874586:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790450048:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828295.874597:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790450176:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828295.874601:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790450688/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828295.874614:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828295.874616:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828295.874618:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828295.874623:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4932 00000800:00000100:1.0:1787828295.874628:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828295.874628:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828295.874637:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828295.874642:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828295.874649:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828295/real 1787828295] req@ffff9e6804422940 x1874668790450176/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828295.874655:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790450816/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828295.874735:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790450176/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828295.874744:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828295.874748:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828295.874751:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790450176:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828295.874770:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790450304:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828295.874784:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828295.874791:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828295/real 1787828295] req@ffff9e6804422d00 x1874668790450304/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828295.874799:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790450304/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828295.874805:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828295.874807:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828295.874809:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790450304:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828295.874812:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790450432:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828295.874822:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828295.874828:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828295/real 1787828295] req@ffff9e6804420780 x1874668790450432/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828295.874835:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790450432/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828295.874841:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828295.874843:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828295.874845:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790450432:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828295.874848:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790450560:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828295.874858:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828295.874863:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828295/real 1787828295] req@ffff9e6804421680 x1874668790450560/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828295.874875:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790450560/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828295.874880:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828295.874881:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828295.874883:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790450560:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828295.874887:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790450688:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828295.874896:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828295.874902:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828295/real 1787828295] req@ffff9e6804422580 x1874668790450688/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828295.874908:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790450688/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828295.874913:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828295.874915:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828295.874917:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790450688:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828295.874920:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790450816:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828295.874930:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828295.874935:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828295/real 1787828295] req@ffff9e6804420b40 x1874668790450816/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828295.874942:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790450816/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828295.874950:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828295.874952:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828295.874954:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790450816:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828296.385785:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828296.577903:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828297.409809:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828297.601853:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828298.433811:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828298.625855:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828299.457767:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828299.649893:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828300.481818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828300.673863:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828300.993793:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828300.993805:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828300.993809:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828300.993819:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4937 00000100:00080000:0.0:1787828300.993831:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828300.993840:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828300.993879:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790450944/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828300.993912:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828300.993916:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828300.993918:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828300.993923:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4937 00000100:00080000:0.0:1787828300.993927:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828300.993932:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828300.993949:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828300.993964:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790450944:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787828300.993966:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790451072/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828300.993982:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828300.993985:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828300.993987:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828300.993992:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4937 00000100:00080000:0.0:1787828300.993997:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828300.994001:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828300.994006:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828300.994021:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790451200/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828300.994032:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828300.994036:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828300.994038:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828300.994043:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4937 00000100:00080000:0.0:1787828300.994049:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828300.994054:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828300.994061:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828300.994070:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790451072:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828300.994077:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790451328/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828300.994093:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828300.994097:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828300.994100:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828300.994103:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790451200:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828300.994109:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4937 00000100:00080000:0.0:1787828300.994116:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828300.994121:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828300.994128:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828300.994130:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790451328:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828300.994151:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790451456/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828300.994171:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828300.994176:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787828300.994177:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790451456:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787828300.994179:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828300.994187:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4937 00000100:00080000:0.0:1787828300.994194:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828300.994201:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828300.994207:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828300.994223:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790451584/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828300.994242:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828300.994246:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828300.994248:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828300.994252:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4937 00000100:00080000:0.0:1787828300.994257:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828300.994262:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828300.994267:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790451584:192.168.203.109@tcp:38:kworker.0 00000008:00100000:0.0:1787828300.994270:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828300.994289:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790451712/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828300.994307:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828300.994311:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828300.994313:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828300.994319:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4937 00000100:00080000:0.0:1787828300.994324:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828300.994325:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790451712:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828300.994332:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828300.994339:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828300.994356:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790451840/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828300.994373:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828300.994377:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828300.994380:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828300.994384:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790451840:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787828300.994390:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4937 00000100:00080000:0.0:1787828300.994396:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828300.994401:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828300.994407:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828300.994422:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790451968/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828300.994451:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790451968:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828301.505773:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828301.697844:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828302.529819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828302.721877:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828303.553775:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828303.745807:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787828304.065928:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:2.0:1787828304.065972:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787828304.065990:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787828304.066044:0:2387:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:2.0:1787828304.066058:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787828304.066068:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787828304.066080:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787828304.066095:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:1.0:1787828304.066097:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828300/real 1787828304] req@ffff9e6803e99e00 x1874668790451968/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787828304.066107:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787828304.066121:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:1.0:1787828304.066133:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790451968/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787828304.066135:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00080000:1.0:1787828304.066152:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828304.066158:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828304.066163:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790451968:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828304.066171:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828300/real 1787828304] req@ffff9e6803e9a940 x1874668790451840/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828304.066186:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790451840/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828304.066196:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828304.066200:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828304.066203:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790451840:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828304.066210:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828300/real 1787828304] req@ffff9e6803e9b840 x1874668790451712/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828304.066222:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790451712/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828304.066231:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828304.066234:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828304.066237:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790451712:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828304.066244:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828300/real 1787828304] req@ffff9e6803e98f00 x1874668790451584/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828304.066256:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790451584/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828304.066266:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828304.066269:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828304.066272:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790451584:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828304.066279:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828300/real 1787828304] req@ffff9e6803e9a1c0 x1874668790451456/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828304.066293:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790451456/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828304.066301:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828304.066304:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828304.066307:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790451456:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828304.066314:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828300/real 1787828304] req@ffff9e6803e98780 x1874668790451328/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828304.066326:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790451328/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828304.066335:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828304.066339:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828304.066342:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790451328:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828304.066349:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828300/real 1787828304] req@ffff9e6803e99680 x1874668790451072/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828304.066360:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790451072/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828304.066369:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828304.066372:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828304.066376:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790451072:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828304.066383:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828300/real 1787828304] req@ffff9e6803e9b480 x1874668790451200/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828304.066393:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790451200/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828304.066401:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828304.066405:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828304.066408:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790451200:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828304.066414:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828300/real 1787828304] req@ffff9e6803e9a580 x1874668790450944/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828304.066426:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790450944/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828304.066434:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828304.066438:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828304.066441:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790450944:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828304.577788:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828304.769858:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828305.601792:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828305.793907:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828306.113884:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828306.113894:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828306.113899:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828306.113909:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4942 00000100:00080000:2.0:1787828306.113918:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828306.113926:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828306.113958:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790452096/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828306.113988:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828306.113991:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828306.113993:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828306.113997:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4942 00000100:00080000:2.0:1787828306.114002:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828306.114006:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828306.114015:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828306.114026:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790452224/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828306.114037:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828306.114040:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828306.114043:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828306.114047:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4942 00000100:00080000:2.0:1787828306.114051:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828306.114054:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828306.114059:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828306.114080:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790452096:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787828306.114080:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790452352/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828306.114102:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828306.114107:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828306.114111:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828306.114116:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4942 00000100:00080000:2.0:1787828306.114120:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828306.114125:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828306.114131:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828306.114145:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790452480/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828306.114162:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828306.114166:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828306.114167:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828306.114171:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4942 00000100:00080000:2.0:1787828306.114176:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828306.114184:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1787828306.114187:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:2.0:1787828306.114193:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828306.114209:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790452608/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828306.114220:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828306.114224:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828306.114226:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828306.114231:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4942 00000100:00080000:2.0:1787828306.114237:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828306.114243:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828306.114249:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828306.114259:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828306/real 1787828306] req@ffff9e6804423480 x1874668790452096/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828306.114267:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790452736/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828306.114309:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828306.114313:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828306.114315:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828306.114323:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4942 00000100:00100000:1.0:1787828306.114326:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790452096/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828306.114329:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828306.114341:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828306.114354:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787828306.114362:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787828306.114371:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790452864/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828306.114373:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828306.114389:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790452096:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828306.114403:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828306.114408:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828306.114411:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828306.114417:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790452224:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828306.114423:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4942 00000100:00080000:2.0:1787828306.114435:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828306.114441:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828306.114448:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828306.114464:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790452992/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828306.114476:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787828306.114478:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828306.114501:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828306.114504:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828306.114511:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4942 00000100:00080000:2.0:1787828306.114517:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828306.114522:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828306/real 1787828306] req@ffff9e68044212c0 x1874668790452224/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828306.114527:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828306.114539:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828306.114567:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790452224/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828306.114569:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790453120/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828306.114654:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828306.114667:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828306.114674:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790452224:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828306.114749:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790452352:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828306.114779:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828306.114793:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828306/real 1787828306] req@ffff9e68044230c0 x1874668790452352/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828306.114805:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790452352/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828306.114814:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828306.114818:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828306.114821:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790452352:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828306.114828:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790452480:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828306.114851:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828306.114864:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828306/real 1787828306] req@ffff9e68044221c0 x1874668790452480/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828306.114875:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790452480/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828306.114884:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828306.114888:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828306.114892:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790452480:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828306.114899:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790452608:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828306.114921:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828306.114933:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828306/real 1787828306] req@ffff9e6804423840 x1874668790452608/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828306.114951:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790452608/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828306.114960:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828306.114963:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828306.114966:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790452608:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828306.114973:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790452736:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828306.114995:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828306.115008:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828306/real 1787828306] req@ffff9e6804420b40 x1874668790452736/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828306.115021:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790452736/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828306.115032:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828306.115036:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828306.115039:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790452736:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828306.115046:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790452864:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828306.115067:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828306.115079:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828306/real 1787828306] req@ffff9e6804422580 x1874668790452864/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828306.115090:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790452864/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828306.115103:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828306.115106:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828306.115109:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790452864:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828306.115115:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790452992:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828306.115136:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828306.115147:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828306/real 1787828306] req@ffff9e6804421680 x1874668790452992/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828306.115158:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790452992/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828306.115167:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828306.115170:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828306.115174:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790452992:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828306.115181:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790453120:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828306.115203:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828306.115215:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828306/real 1787828306] req@ffff9e6804420780 x1874668790453120/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828306.115228:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790453120/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828306.115237:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828306.115241:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828306.115245:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790453120:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828306.625782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828306.817847:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828307.649776:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828307.841848:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828308.673778:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828308.865892:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828309.697757:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828309.889952:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828310.721793:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828310.913823:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828311.233833:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828311.233842:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828311.233845:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828311.233853:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4947 00000100:00080000:3.0:1787828311.233862:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828311.233867:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828311.233900:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790453248/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828311.233920:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828311.233923:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828311.233925:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828311.233928:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4947 00000100:00080000:3.0:1787828311.233930:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828311.233933:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828311.233941:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828311.233952:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790453376/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828311.233954:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790453248:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828311.233967:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828311.233971:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828311.233972:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828311.233975:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4947 00000100:00080000:3.0:1787828311.233977:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828311.233979:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828311.233983:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828311.233992:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790453504/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828311.233999:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828311.234002:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828311.234003:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828311.234005:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4947 00000100:00080000:3.0:1787828311.234008:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828311.234010:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828311.234014:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828311.234018:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828311.234026:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790453632/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828311.234035:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828311.234037:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828311.234038:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828311.234041:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4947 00000100:00080000:3.0:1787828311.234043:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828311.234046:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828311.234049:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828311.234076:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828311/real 1787828311] req@ffff9e680a2d61c0 x1874668790453248/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828311.234080:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790453760/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828311.234120:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828311.234124:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828311.234126:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828311.234130:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4947 00000100:00080000:3.0:1787828311.234134:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828311.234139:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790453248/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828311.234142:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828311.234148:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828311.234174:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790453888/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828311.234193:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828311.234201:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828311.234209:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790453248:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828311.234211:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828311.234218:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828311.234221:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828311.234226:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4947 00000100:00080000:3.0:1787828311.234232:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828311.234238:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790453376:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828311.234239:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828311.234249:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828311.234264:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790454016/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828311.234273:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787828311.234279:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828311.234279:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828311.234284:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828311.234289:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4947 00000100:00080000:3.0:1787828311.234293:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828311.234296:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828311.234300:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828311.234304:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828311/real 1787828311] req@ffff9e680a2d43c0 x1874668790453376/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828311.234311:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790454144/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828311.234354:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828311.234359:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787828311.234361:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790453376/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828311.234362:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828311.234371:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4947 00000100:00080000:3.0:1787828311.234379:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828311.234404:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787828311.234405:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828311.234410:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828311.234414:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828311.234421:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790453376:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828311.234426:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790454272/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828311.234434:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790453504:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828311.234517:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828311.234537:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828311/real 1787828311] req@ffff9e680a2d5a40 x1874668790453504/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828311.234549:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790453504/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828311.234556:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828311.234559:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828311.234562:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790453504:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828311.234567:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790453632:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828311.234583:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828311.234591:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828311/real 1787828311] req@ffff9e680a2d4000 x1874668790453632/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828311.234600:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790453632/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828311.234607:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828311.234609:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828311.234611:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790453632:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828311.234616:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790453760:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828311.234629:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828311.234637:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828311/real 1787828311] req@ffff9e680a2d70c0 x1874668790453760/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828311.234646:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790453760/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828311.234668:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828311.234670:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828311.234671:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790453760:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828311.234675:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790453888:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828311.234685:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828311.234691:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828311/real 1787828311] req@ffff9e680a2d6580 x1874668790453888/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828311.234698:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790453888/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828311.234705:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828311.234706:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828311.234708:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790453888:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828311.234741:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790454016:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828311.234753:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828311.234760:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828311/real 1787828311] req@ffff9e680a2d61c0 x1874668790454016/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828311.234767:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790454016/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828311.234773:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828311.234775:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828311.234777:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790454016:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828311.234780:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790454144:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828311.234789:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828311.234795:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828311/real 1787828311] req@ffff9e680a2d4b40 x1874668790454144/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828311.234801:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790454144/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828311.234806:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828311.234808:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828311.234810:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790454144:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828311.234813:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790454272:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828311.234822:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828311.234827:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828311/real 1787828311] req@ffff9e680a2d52c0 x1874668790454272/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828311.234833:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790454272/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828311.234838:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828311.234840:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828311.234842:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790454272:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828311.745787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828311.937840:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828312.769835:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828312.961870:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828313.793788:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828313.985905:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828314.817778:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828315.009914:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828315.841807:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828316.033957:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828316.353851:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828316.353864:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828316.353870:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828316.353883:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4953 00000100:00080000:3.0:1787828316.353896:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828316.353904:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828316.353951:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790454400/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828316.353985:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828316.353989:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828316.353992:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828316.353997:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4953 00000100:00080000:3.0:1787828316.354000:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828316.354004:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828316.354016:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828316.354032:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790454528/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828316.354046:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828316.354051:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790454400:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828316.354052:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828316.354059:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828316.354070:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4953 00000100:00080000:3.0:1787828316.354073:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828316.354077:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828316.354083:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828316.354097:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790454656/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828316.354110:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828316.354114:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828316.354116:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828316.354121:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4953 00000100:00080000:3.0:1787828316.354126:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828316.354130:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828316.354136:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828316.354150:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790454784/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828316.354160:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828316.354171:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828316.354176:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828316.354180:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828316.354187:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4953 00000100:00080000:3.0:1787828316.354192:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828316.354197:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828316.354202:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828316.354212:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790454912/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828316.354223:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828316.354227:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828316.354229:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828316.354235:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4953 00000100:00080000:3.0:1787828316.354240:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828316.354248:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828316/real 1787828316] req@ffff9e680a2d4f00 x1874668790454400/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828316.354250:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828316.354264:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828316.354290:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790455040/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828316.354296:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790454400/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828316.354330:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828316.354336:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828316.354337:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828316.354344:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828316.354350:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828316.354358:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4953 00000100:00100000:1.0:1787828316.354364:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790454400:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828316.354367:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828316.354381:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828316.354387:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828316.354400:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790455168/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828316.354434:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790454528:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828316.354450:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828316.354456:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828316.354458:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828316.354467:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4953 00000100:00080000:3.0:1787828316.354474:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828316.354485:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828316.354485:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828316.354500:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828316.354515:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828316/real 1787828316] req@ffff9e680a2d6d00 x1874668790454528/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828316.354524:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790455296/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828316.354552:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828316.354554:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790454528/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828316.354557:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828316.354564:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828316.354580:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4953 00000100:00080000:1.0:1787828316.354585:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828316.354588:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828316.354599:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828316.354601:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828316.354615:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790454528:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787828316.354615:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828316.354639:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790455424/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828316.354641:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790454656:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828316.354740:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828316.354764:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828316/real 1787828316] req@ffff9e680a2d4780 x1874668790454656/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828316.354788:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790454656/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828316.354804:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828316.354810:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828316.354817:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790454656:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828316.354830:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790454784:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828316.354870:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828316.354891:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828316/real 1787828316] req@ffff9e680a2d5e00 x1874668790454784/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828316.354915:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790454784/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828316.354932:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828316.354938:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828316.354945:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790454784:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828316.354953:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790454912:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828316.354981:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828316.354996:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828316/real 1787828316] req@ffff9e680a2d7480 x1874668790454912/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828316.355009:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790454912/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828316.355019:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828316.355023:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828316.355027:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790454912:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828316.355033:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790455040:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828316.355051:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828316.355061:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828316/real 1787828316] req@ffff9e680a2d7c00 x1874668790455040/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828316.355074:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790455040/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828316.355085:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828316.355088:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828316.355091:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790455040:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828316.355117:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790455168:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828316.355142:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828316.355162:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828316/real 1787828316] req@ffff9e680a2d52c0 x1874668790455168/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828316.355185:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790455168/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828316.355201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828316.355208:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828316.355214:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790455168:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828316.355225:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790455296:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828316.355272:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828316.355288:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828316/real 1787828316] req@ffff9e680a2d4b40 x1874668790455296/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828316.355302:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790455296/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828316.355313:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828316.355317:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828316.355320:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790455296:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828316.355327:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790455424:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828316.355347:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828316.355358:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828316/real 1787828316] req@ffff9e680a2d61c0 x1874668790455424/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828316.355371:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790455424/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828316.355381:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828316.355384:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828316.355406:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790455424:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828316.865856:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828317.057929:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828317.889767:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828318.081965:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828318.913847:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828319.105937:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828319.937826:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828320.129875:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828320.961841:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828321.153994:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828321.473849:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828321.473863:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828321.473869:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828321.473883:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4958 00000100:00080000:3.0:1787828321.473895:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828321.473906:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828321.473974:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790455552/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828321.474016:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828321.474021:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828321.474024:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828321.474029:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4958 00000100:00080000:3.0:1787828321.474035:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828321.474040:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828321.474064:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828321.474085:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790455680/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828321.474101:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828321.474109:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828321.474113:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828321.474114:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790455552:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828321.474127:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4958 00000100:00080000:3.0:1787828321.474132:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828321.474137:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828321.474143:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828321.474160:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790455808/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828321.474179:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828321.474185:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828321.474206:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828321.474214:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4958 00000100:00080000:3.0:1787828321.474222:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828321.474229:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828321.474236:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828321.474251:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828321.474280:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790455936/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828321.474296:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828321/real 1787828321] req@ffff9e680a2d6580 x1874668790455552/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828321.474304:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828321.474325:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828321.474333:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828321.474346:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4958 00000100:00080000:3.0:1787828321.474355:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828321.474360:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790455552/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828321.474370:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787828321.474417:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828321.474423:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828321.474430:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828321.474439:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790455552:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828321.474453:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790456064/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828321.474467:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828321.474471:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828321.474474:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828321.474477:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790455680:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828321.474492:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4958 00000100:00080000:3.0:1787828321.474502:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787828321.474513:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828321.474526:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828321/real 1787828321] req@ffff9e680a2d70c0 x1874668790455680/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828321.474529:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828321.474554:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790455680/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828321.474554:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787828321.474580:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828321.474587:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828321.474593:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790456192/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828321.474604:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790455680:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828321.474614:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828321.474620:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828321.474626:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828321.474627:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790455808:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828321.474645:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4958 00000100:00080000:3.0:1787828321.474653:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828321.474658:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828321.474665:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828321.474675:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828321.474682:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790456320/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828321.474731:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828321/real 1787828321] req@ffff9e680a2d4000 x1874668790455808/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828321.474755:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828321.474760:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828321.474763:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828321.474766:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790455808/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828321.474774:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4958 00000100:00080000:3.0:1787828321.474786:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828321.474796:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828321.474797:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787828321.474815:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:3.0:1787828321.474817:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828321.474824:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790455808:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828321.474835:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790455936:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828321.474842:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790456448/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828321.474856:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828321.474861:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787828321.474867:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828321.474869:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828321.474879:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4958 00000100:00080000:3.0:1787828321.474885:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828321.474890:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828321/real 1787828321] req@ffff9e680a2d5a40 x1874668790455936/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828321.474896:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828321.474920:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828321.474924:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790455936/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828321.474937:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828321.474940:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828321.474947:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790455936:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828321.474956:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790456576/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828321.474958:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790456064:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828321.474988:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828321.475003:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828321/real 1787828321] req@ffff9e680a2d43c0 x1874668790456064/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828321.475013:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790456064/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828321.475019:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828321.475023:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828321.475027:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790456064:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828321.475047:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790456192:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828321.475059:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828321.475066:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828321/real 1787828321] req@ffff9e680a2d6580 x1874668790456192/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828321.475074:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790456192/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828321.475081:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828321.475083:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828321.475085:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790456192:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828321.475088:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790456320:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828321.475099:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828321.475104:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828321/real 1787828321] req@ffff9e680a2d61c0 x1874668790456320/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828321.475112:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790456320/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828321.475118:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828321.475120:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828321.475123:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790456320:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828321.475127:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790456448:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828321.475137:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828321.475142:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828321/real 1787828321] req@ffff9e680a2d4b40 x1874668790456448/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828321.475150:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790456448/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828321.475155:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828321.475157:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828321.475159:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790456448:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828321.475163:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790456576:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828321.475173:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828321.475178:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828321/real 1787828321] req@ffff9e680a2d52c0 x1874668790456576/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828321.475185:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790456576/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828321.475190:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828321.475203:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828321.475206:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790456576:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828321.985854:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828322.177953:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828323.009825:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828323.201893:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828324.033827:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828324.226056:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828325.057835:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828325.249936:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828326.081798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828326.273842:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828326.593768:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828326.593776:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828326.593780:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828326.593788:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4963 00000100:00080000:3.0:1787828326.593798:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828326.593804:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828326.593835:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790456704/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828326.593857:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828326.593860:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828326.593862:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828326.593865:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4963 00000100:00080000:3.0:1787828326.593867:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828326.593870:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828326.593878:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828326.593887:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790456832/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828326.593894:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828326.593897:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828326.593899:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828326.593902:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790456704:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828326.593903:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4963 00000100:00080000:3.0:1787828326.593908:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828326.593914:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828326.593918:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828326.593926:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790456960/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828326.593933:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828326.593935:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828326.593937:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828326.593940:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4963 00000100:00080000:3.0:1787828326.593942:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828326.593944:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828326.593947:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828326.593955:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790457088/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828326.593961:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828326.593963:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828326.593964:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828326.593968:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828326.593970:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4963 00000100:00080000:3.0:1787828326.593975:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828326.593978:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828326.593982:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828326.593989:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790457216/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828326.593995:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828326.593997:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828326.593999:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828326.594002:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4963 00000100:00080000:3.0:1787828326.594006:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828326.594009:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828326/real 1787828326] req@ffff9e680a2d7c00 x1874668790456704/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828326.594013:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828326.594020:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828326.594034:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790457344/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828326.594038:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790456704/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828326.594054:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828326.594058:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828326.594060:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828326.594061:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828326.594066:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828326.594072:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4963 00000100:00100000:1.0:1787828326.594073:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790456704:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828326.594079:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828326.594085:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828326.594090:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828326.594106:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790457472/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828326.594111:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790456832:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828326.594127:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828326.594129:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828326.594131:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828326.594135:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4963 00000800:00000100:1.0:1787828326.594139:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828326.594142:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828326.594148:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828326.594154:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828326/real 1787828326] req@ffff9e680a2d7480 x1874668790456832/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828326.594154:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828326.594170:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790456832/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828326.594170:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790457600/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828326.594203:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828326.594203:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828326.594211:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828326.594212:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828326.594216:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828326.594218:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790456832:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828326.594222:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4963 00000100:00080000:3.0:1787828326.594226:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828326.594227:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790456960:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828326.594232:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828326.594237:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828326.594248:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790457728/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828326.594251:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828326.594268:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828326/real 1787828326] req@ffff9e680a2d5e00 x1874668790456960/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828326.594277:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790456960/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828326.594283:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828326.594285:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828326.594288:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790456960:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828326.594291:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790457088:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828326.594302:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828326.594308:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828326/real 1787828326] req@ffff9e680a2d4780 x1874668790457088/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828326.594315:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790457088/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828326.594320:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828326.594322:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828326.594324:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790457088:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828326.594327:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790457216:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828326.594337:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828326.594363:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828326/real 1787828326] req@ffff9e680a2d6d00 x1874668790457216/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828326.594371:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790457216/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828326.594377:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828326.594379:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828326.594392:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790457216:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828326.594396:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790457344:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828326.594410:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828326.594416:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828326/real 1787828326] req@ffff9e680a2d4f00 x1874668790457344/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828326.594423:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790457344/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828326.594429:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828326.594431:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828326.594433:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790457344:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828326.594447:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790457472:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828326.594458:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828326.594464:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828326/real 1787828326] req@ffff9e680a2d52c0 x1874668790457472/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828326.594476:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790457472/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828326.594481:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828326.594483:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828326.594484:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790457472:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828326.594488:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790457600:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828326.594498:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828326.594503:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828326/real 1787828326] req@ffff9e680a2d4b40 x1874668790457600/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828326.594510:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790457600/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828326.594515:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828326.594516:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828326.594518:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790457600:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828326.594521:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790457728:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828326.594530:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828326.594535:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828326/real 1787828326] req@ffff9e680a2d61c0 x1874668790457728/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828326.594542:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790457728/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828326.594551:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828326.594553:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828326.594556:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790457728:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828327.105770:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828327.297927:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828328.129764:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828328.321841:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828329.153799:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828329.345936:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828330.177787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828330.369886:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828331.201870:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828331.393999:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828331.713934:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828331.713955:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828331.713964:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828331.713980:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4968 00000100:00080000:1.0:1787828331.713999:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828331.714010:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828331.714073:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790457856/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828331.714121:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828331.714126:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828331.714129:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828331.714135:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4968 00000100:00080000:1.0:1787828331.714140:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828331.714146:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828331.714163:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828331.714191:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790457984/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828331.714191:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790457856:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828331.714227:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828331.714233:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828331.714236:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828331.714246:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4968 00000100:00080000:1.0:1787828331.714251:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828331.714257:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828331.714264:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828331.714279:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790458112/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828331.714293:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828331.714325:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828331.714338:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828331.714342:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828331.714354:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4968 00000100:00080000:1.0:1787828331.714360:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828331.714366:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828331.714377:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828331.714392:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828331/real 1787828331] req@ffff9e6831aaad00 x1874668790457856/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828331.714400:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790458240/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828331.714461:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828331.714470:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828331.714471:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790457856/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828331.714476:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828331.714500:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4968 00000100:00080000:1.0:1787828331.714513:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828331.714513:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828331.714535:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787828331.714535:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:1.0:1787828331.714559:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828331.714560:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790457856:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828331.714592:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790458368/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828331.714614:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828331.714619:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828331.714622:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828331.714626:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790457984:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828331.714633:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4968 00000100:00080000:1.0:1787828331.714652:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828331.714663:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828331.714671:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828331.714688:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828331.714761:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828331/real 1787828331] req@ffff9e6831aa8000 x1874668790457984/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828331.714764:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790458496/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828331.714865:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828331.714872:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790457984/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828331.714876:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828331.714885:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828331.714909:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4968 00000100:00080000:2.0:1787828331.714917:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828331.714922:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828331.714931:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828331.714941:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828331.714953:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790457984:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787828331.714958:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828331.714983:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790458112:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828331.714990:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790458624/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828331.715009:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828331.715014:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828331.715017:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828331.715024:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4968 00000100:00080000:1.0:1787828331.715033:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828331.715044:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828331.715046:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828331.715058:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828331.715078:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790458752/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828331.715090:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828331/real 1787828331] req@ffff9e6831aa9a40 x1874668790458112/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828331.715097:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828331.715109:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828331.715115:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828331.715138:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4968 00000100:00080000:1.0:1787828331.715154:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828331.715162:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828331.715168:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828331.715174:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790458112/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828331.715186:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790458880/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828331.715240:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828331.715250:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828331.715258:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790458112:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828331.715270:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790458240:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828331.715330:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828331.715347:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828331/real 1787828331] req@ffff9e6831aab480 x1874668790458240/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828331.715361:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790458240/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828331.715370:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828331.715374:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828331.715377:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790458240:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828331.715398:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790458368:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828331.715421:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828331.715431:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828331/real 1787828331] req@ffff9e6831aabc00 x1874668790458368/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828331.715442:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790458368/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828331.715451:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828331.715455:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828331.715458:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790458368:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828331.715464:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790458496:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828331.715479:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828331.715488:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828331/real 1787828331] req@ffff9e6831aaad00 x1874668790458496/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828331.715499:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790458496/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828331.715509:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828331.715512:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828331.715515:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790458496:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828331.715521:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790458624:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828331.715536:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828331.715545:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828331/real 1787828331] req@ffff9e6831aaa1c0 x1874668790458624/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828331.715556:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790458624/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828331.715564:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828331.715567:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828331.715570:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790458624:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828331.715577:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790458752:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828331.715593:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828331.715602:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828331/real 1787828331] req@ffff9e6831aa8780 x1874668790458752/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828331.715613:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790458752/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828331.715620:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828331.715623:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828331.715626:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790458752:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828331.715632:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790458880:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828331.715647:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828331.715656:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828331/real 1787828331] req@ffff9e6831aaa940 x1874668790458880/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828331.715667:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790458880/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828331.715675:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828331.715678:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828331.715680:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790458880:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828332.225847:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828332.417921:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828333.249830:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828333.442000:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828334.273796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828334.465843:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828335.297824:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828335.489887:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828336.321813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828336.513953:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828336.833806:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828336.833818:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828336.833824:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828336.833836:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4973 00000100:00080000:1.0:1787828336.833848:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828336.833857:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828336.833925:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790459008/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828336.833973:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828336.833979:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828336.833982:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828336.833987:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4973 00000100:00080000:1.0:1787828336.834012:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828336.834023:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828336.834040:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828336.834057:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790459008:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828336.834077:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790459136/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828336.834107:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828336.834113:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828336.834117:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828336.834123:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4973 00000100:00080000:1.0:1787828336.834150:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828336.834157:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828336.834168:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828336.834170:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828336.834198:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790459264/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828336.834216:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828336.834221:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828336.834224:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828336.834230:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4973 00000100:00080000:1.0:1787828336.834237:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828336.834244:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828336.834251:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828336.834259:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828336/real 1787828336] req@ffff9e6831aa9e00 x1874668790459008/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828336.834303:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790459392/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828336.834331:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828336.834341:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790459008/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828336.834341:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828336.834350:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828336.834367:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4973 00000100:00080000:1.0:1787828336.834381:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828336.834391:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787828336.834394:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787828336.834400:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787828336.834411:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828336.834425:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790459520/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828336.834429:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790459008:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828336.834457:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828336.834461:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828336.834465:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828336.834472:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4973 00000100:00080000:1.0:1787828336.834476:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828336.834481:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828336.834488:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828336.834491:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790459136:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828336.834509:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790459648/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828336.834526:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828336.834531:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828336.834534:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828336.834540:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4973 00000800:00000100:0.0:1787828336.834549:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828336.834552:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828336.834563:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828336.834573:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828336.834587:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828336/real 1787828336] req@ffff9e6831aa8f00 x1874668790459136/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828336.834603:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790459776/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828336.834634:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828336.834643:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790459136/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828336.834645:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828336.834654:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828336.834674:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4973 00000100:00080000:1.0:1787828336.834685:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828336.834689:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828336.834693:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828336.834748:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828336.834772:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790459904/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828336.834794:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828336.834796:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828336.834808:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828336.834814:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828336.834817:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790459136:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828336.834827:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4973 00000100:00080000:1.0:1787828336.834836:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828336.834846:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828336.834849:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790459264:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787828336.834855:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828336.834878:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790460032/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828336.834913:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828336.834945:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828336/real 1787828336] req@ffff9e6831aaa580 x1874668790459264/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828336.834962:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790459264/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828336.834973:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828336.834977:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828336.834982:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790459264:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828336.834990:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790459392:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828336.835012:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828336.835023:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828336/real 1787828336] req@ffff9e6831aa9680 x1874668790459392/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828336.835037:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790459392/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828336.835048:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828336.835051:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828336.835055:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790459392:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828336.835062:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790459520:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828336.835081:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828336.835092:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828336/real 1787828336] req@ffff9e6831aa8b40 x1874668790459520/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828336.835106:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790459520/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828336.835116:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828336.835119:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828336.835123:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790459520:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828336.835146:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790459648:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828336.835165:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828336.835177:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828336/real 1787828336] req@ffff9e6831aa9e00 x1874668790459648/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828336.835191:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790459648/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828336.835203:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828336.835207:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828336.835210:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790459648:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828336.835217:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790459776:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828336.835236:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828336.835246:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828336/real 1787828336] req@ffff9e6831aaa940 x1874668790459776/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828336.835301:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790459776/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828336.835312:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828336.835316:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828336.835320:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790459776:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828336.835328:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790459904:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828336.835355:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828336.835369:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828336/real 1787828336] req@ffff9e6831aa8780 x1874668790459904/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828336.835384:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790459904/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828336.835393:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828336.835397:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828336.835400:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790459904:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828336.835407:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790460032:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828336.835425:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828336.835436:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828336/real 1787828336] req@ffff9e6831aaa1c0 x1874668790460032/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828336.835449:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790460032/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828336.835467:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828336.835471:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828336.835474:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790460032:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828337.345836:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828337.538006:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828338.369841:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828338.561924:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828339.393844:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828339.585892:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828340.417798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828340.609887:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828341.441850:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828341.633949:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828341.953850:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828341.953860:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828341.953865:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828341.953875:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4978 00000100:00080000:3.0:1787828341.953886:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828341.953894:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828341.953954:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790460160/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828341.953999:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828341.954005:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828341.954009:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828341.954016:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4978 00000100:00080000:3.0:1787828341.954022:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828341.954030:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828341.954042:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828341.954063:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790460160:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828341.954064:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790460288/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828341.954103:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828341.954106:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828341.954108:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828341.954113:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4978 00000100:00080000:3.0:1787828341.954117:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828341.954121:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828341.954126:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828341.954139:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790460416/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828341.954149:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828341.954153:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828341.954155:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828341.954161:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4978 00000800:00000100:0.0:1787828341.954163:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828341.954166:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828341.954172:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828341.954177:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828341.954189:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790460544/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828341.954198:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828341.954202:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828341.954204:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828341.954212:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4978 00000100:00100000:0.0:1787828341.954214:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828341/real 1787828341] req@ffff9e680a2d6580 x1874668790460160/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828341.954216:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828341.954224:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828341.954254:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828341.954276:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790460672/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828341.954279:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790460160/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828341.954322:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828341.954329:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828341.954334:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828341.954334:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828341.954351:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4978 00000100:00080000:0.0:1787828341.954352:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828341.954355:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828341.954367:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828341.954369:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790460160:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787828341.954373:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828341.954394:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790460800/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828341.954404:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828341.954407:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828341.954410:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828341.954434:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790460288:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828341.954438:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4978 00000100:00080000:3.0:1787828341.954447:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828341.954455:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828341.954461:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828341.954476:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790460928/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828341.954513:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787828341.954519:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828341.954522:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828341.954528:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828341.954536:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4978 00000100:00080000:3.0:1787828341.954543:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828341.954565:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828341/real 1787828341] req@ffff9e680a2d43c0 x1874668790460288/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828341.954573:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828341.954589:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828341.954597:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790460288/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828341.954611:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828341.954612:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790461056/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828341.954622:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828341.954649:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790460288:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828341.954660:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828341.954662:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790460416:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828341.954675:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828341.954679:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828341.954687:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4978 00000100:00080000:3.0:1787828341.954695:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828341.954697:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828341.954710:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828341.954717:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828341/real 1787828341] req@ffff9e680a2d5a40 x1874668790460416/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828341.954720:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828341.954737:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790460416/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828341.954744:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790461184/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828341.954757:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828341.954768:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828341.954822:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790460416:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828341.954831:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790460544:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828341.954855:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828341.954867:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828341/real 1787828341] req@ffff9e680a2d4000 x1874668790460544/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828341.954879:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790460544/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828341.954887:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828341.954890:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828341.954893:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790460544:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828341.954900:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790460672:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828341.954914:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828341.954946:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828341/real 1787828341] req@ffff9e680a2d70c0 x1874668790460672/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828341.954959:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790460672/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828341.954968:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828341.954971:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828341.954974:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790460672:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828341.954980:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790460800:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828341.954996:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828341.955005:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828341/real 1787828341] req@ffff9e680a2d61c0 x1874668790460800/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828341.955016:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790460800/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828341.955025:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828341.955028:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828341.955031:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790460800:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828341.955053:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790460928:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828341.955069:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828341.955078:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828341/real 1787828341] req@ffff9e680a2d4b40 x1874668790460928/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828341.955105:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790460928/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828341.955120:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828341.955123:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828341.955126:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790460928:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828341.955131:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790461056:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828341.955147:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828341.955155:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828341/real 1787828341] req@ffff9e680a2d52c0 x1874668790461056/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828341.955166:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790461056/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828341.955173:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828341.955176:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828341.955178:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790461056:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828341.955183:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790461184:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828341.955196:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828341.955204:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828341/real 1787828341] req@ffff9e680a2d4f00 x1874668790461184/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828341.955214:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790461184/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828366 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828341.955221:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828341.955224:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828341.955227:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790461184:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828342.465836:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828342.657908:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828343.489852:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828343.682073:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828344.513783:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828344.706006:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828345.537868:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828345.729929:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828346.561839:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828346.753952:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828347.073878:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828347.073890:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828347.073896:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828347.073918:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4983 00000100:00080000:2.0:1787828347.073931:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828347.073939:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828347.073986:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790461312/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828347.074033:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828347.074038:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828347.074041:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828347.074046:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4983 00000100:00080000:2.0:1787828347.074050:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828347.074054:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828347.074066:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828347.074082:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790461440/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828347.074094:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828347.074099:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828347.074101:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828347.074105:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4983 00000100:00080000:2.0:1787828347.074109:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828347.074113:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828347.074119:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828347.074125:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790461312:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787828347.074135:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790461568/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828347.074158:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828347.074163:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828347.074165:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828347.074171:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4983 00000100:00080000:2.0:1787828347.074176:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828347.074180:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828347.074186:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828347.074218:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790461696/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828347.074229:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828347.074234:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828347.074241:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787828347.074242:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828347.074254:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4983 00000100:00080000:2.0:1787828347.074267:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828347.074273:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828347.074279:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828347.074295:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790461824/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828347.074307:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828347.074311:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828347.074314:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828347.074322:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828347/real 1787828347] req@ffff9e6804422d00 x1874668790461312/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828347.074325:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4983 00000100:00080000:2.0:1787828347.074336:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828347.074347:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828347.074356:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828347.074365:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790461312/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828347.074372:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790461952/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828347.074398:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828347.074401:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828347.074408:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828347.074410:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787828347.074413:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828347.074422:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790461312:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828347.074428:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4983 00000100:00080000:2.0:1787828347.074437:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828347.074442:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790461440:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828347.074446:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828347.074457:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828347.074472:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790462080/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828347.074484:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787828347.074493:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828347.074494:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828347.074501:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828347.074510:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4983 00000100:00080000:2.0:1787828347.074516:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828347.074521:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828347/real 1787828347] req@ffff9e6804422940 x1874668790461440/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828347.074527:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828347.074539:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828347.074553:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790461440/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828347.074568:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790462208/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828347.074570:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828347.074584:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828347.074598:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790461440:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828347.074618:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828347.074623:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828347.074626:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828347.074632:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4983 00000100:00080000:2.0:1787828347.074637:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828347.074642:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828347.074647:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790461568:192.168.203.109@tcp:38:kworker.0 00000008:00100000:2.0:1787828347.074652:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828347.074675:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790462336/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828347.074688:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828347.074775:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828347/real 1787828347] req@ffff9e6804421e00 x1874668790461568/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828347.074790:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790461568/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828347.074800:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828347.074804:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828347.074808:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790461568:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828347.074815:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790461696:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828347.074861:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828347.074874:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828347/real 1787828347] req@ffff9e6804420f00 x1874668790461696/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828347.074888:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790461696/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828347.074897:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828347.074901:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828347.074904:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790461696:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828347.074910:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790461824:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828347.074930:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828347.074940:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828347/real 1787828347] req@ffff9e6804420780 x1874668790461824/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828347.074952:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790461824/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828347.074962:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828347.074966:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828347.074969:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790461824:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828347.074975:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790461952:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828347.074992:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828347.075008:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828347/real 1787828347] req@ffff9e6804421680 x1874668790461952/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828347.075029:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790461952/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828347.075065:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828347.075071:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828347.075077:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790461952:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828347.075089:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790462080:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828347.075129:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828347.075147:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828347/real 1787828347] req@ffff9e6804422580 x1874668790462080/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828347.075162:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790462080/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828347.075172:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828347.075176:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828347.075181:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790462080:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828347.075188:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790462208:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828347.075373:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828347.075390:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828347/real 1787828347] req@ffff9e6804420b40 x1874668790462208/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828347.075405:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790462208/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828347.075415:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828347.075419:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828347.075423:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790462208:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828347.075448:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790462336:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828347.075469:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828347.075480:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828347/real 1787828347] req@ffff9e6804423840 x1874668790462336/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828347.075494:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790462336/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828347.075504:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828347.075508:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828347.075511:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790462336:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828347.585835:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828347.777859:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828348.609811:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828348.801872:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828349.633792:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828349.825862:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828350.657835:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828350.849912:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828351.681810:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828351.873880:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828352.193762:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828352.193770:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828352.193774:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828352.193783:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4988 00000100:00080000:2.0:1787828352.193793:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828352.193799:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828352.193828:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790462464/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828352.193849:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828352.193852:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828352.193853:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828352.193856:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4988 00000100:00080000:2.0:1787828352.193858:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828352.193861:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828352.193870:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828352.193880:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790462592/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828352.193885:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790462464:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828352.193891:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828352.193896:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828352.193897:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828352.193900:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4988 00000100:00080000:2.0:1787828352.193902:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828352.193904:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828352.193907:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828352.193916:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790462720/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828352.193930:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828352.193933:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828352.193934:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828352.193936:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4988 00000100:00080000:2.0:1787828352.193939:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828352.193941:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828352.193945:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828352.193955:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828352.193955:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790462848/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828352.193968:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828352.193971:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828352.193973:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828352.193976:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4988 00000100:00080000:2.0:1787828352.193979:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828352.193981:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828352.193985:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828352.193994:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828352/real 1787828352] req@ffff9e68044221c0 x1874668790462464/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828352.193996:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790462976/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828352.194024:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828352.194027:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828352.194028:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828352.194033:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790462464/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828352.194034:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4988 00000100:00080000:2.0:1787828352.194039:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828352.194045:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787828352.194052:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787828352.194052:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787828352.194061:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828352.194064:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790463104/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828352.194067:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790462464:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828352.194080:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828352.194083:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828352.194084:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828352.194088:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4988 00000100:00080000:2.0:1787828352.194090:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828352.194092:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828352.194096:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790462592:192.168.203.109@tcp:38:kworker.0 00000008:00100000:2.0:1787828352.194097:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828352.194110:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790463232/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828352.194118:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828352.194121:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828352.194124:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787828352.194125:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828352.194130:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4988 00000100:00080000:2.0:1787828352.194135:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828352.194138:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828352.194142:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828352/real 1787828352] req@ffff9e68044230c0 x1874668790462592/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828352.194142:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828352.194155:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790463360/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828352.194159:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790462592/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828352.194192:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828352.194195:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828352.194196:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828352.194201:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828352.194202:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787828352.194206:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4988 00000100:00100000:0.0:1787828352.194209:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790462592:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828352.194211:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828352.194218:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790462720:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828352.194218:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828352.194227:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828352.194241:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790463488/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828352.194248:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828352.194261:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828352/real 1787828352] req@ffff9e68044212c0 x1874668790462720/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828352.194270:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790462720/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828352.194280:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828352.194282:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828352.194285:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790462720:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828352.194289:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790462848:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828352.194300:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828352.194305:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828352/real 1787828352] req@ffff9e6804423480 x1874668790462848/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828352.194313:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790462848/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828352.194318:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828352.194320:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828352.194322:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790462848:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828352.194325:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790462976:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828352.194335:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828352.194340:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828352/real 1787828352] req@ffff9e6804423840 x1874668790462976/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828352.194348:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790462976/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828352.194352:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828352.194354:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828352.194364:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790462976:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828352.194368:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790463104:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828352.194380:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828352.194391:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828352/real 1787828352] req@ffff9e6804420b40 x1874668790463104/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828352.194403:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790463104/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828352.194413:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828352.194416:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828352.194420:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790463104:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828352.194438:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790463232:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828352.194454:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828352.194462:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828352/real 1787828352] req@ffff9e6804422580 x1874668790463232/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828352.194477:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790463232/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828352.194483:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828352.194485:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828352.194487:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790463232:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828352.194491:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790463360:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828352.194502:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828352.194508:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828352/real 1787828352] req@ffff9e6804421680 x1874668790463360/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828352.194516:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790463360/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828352.194520:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828352.194522:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828352.194524:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790463360:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828352.194528:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790463488:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828352.194538:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828352.194543:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828352/real 1787828352] req@ffff9e6804420780 x1874668790463488/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828352.194550:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790463488/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828352.194555:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828352.194557:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828352.194558:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790463488:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828352.705833:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828352.898003:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828353.729863:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828353.921957:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828354.753839:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828354.945906:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828355.777820:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828355.969845:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828356.801936:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828356.993999:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828357.313901:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828357.313914:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828357.313919:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828357.313931:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4994 00000100:00080000:0.0:1787828357.313944:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828357.313953:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828357.313999:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790463616/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828357.314050:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828357.314055:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828357.314057:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828357.314062:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4994 00000100:00080000:0.0:1787828357.314067:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828357.314073:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828357.314086:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828357.314102:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790463744/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828357.314111:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790463616:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828357.314163:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828357.314170:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828357.314173:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828357.314180:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4994 00000100:00080000:0.0:1787828357.314187:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828357.314193:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828357.314201:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828357.314221:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790463872/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828357.314232:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828357.314250:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828357.314255:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828357.314258:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828357.314266:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4994 00000100:00080000:0.0:1787828357.314271:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828357.314276:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828357.314285:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828357.314298:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828357/real 1787828357] req@ffff9e6803e992c0 x1874668790463616/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828357.314304:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790464000/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828357.314359:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828357.314364:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828357.314367:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828357.314378:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4994 00000100:00100000:2.0:1787828357.314382:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790463616/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828357.314384:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828357.314394:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828357.314404:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828357.314425:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790464128/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828357.314428:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828357.314451:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787828357.314452:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828357.314464:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787828357.314469:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790463616:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828357.314471:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828357.314488:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4994 00000100:00080000:0.0:1787828357.314493:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828357.314498:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828357.314504:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828357.314517:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790464256/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828357.314524:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790463744:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828357.314550:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828357.314556:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828357.314559:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828357.314566:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4994 00000100:00080000:0.0:1787828357.314572:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828357.314577:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828357.314582:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828357.314595:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790464384/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828357.314610:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828357.314614:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828357.314621:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828357.314624:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828357.314631:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4994 00000100:00080000:0.0:1787828357.314636:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828357.314641:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828357/real 1787828357] req@ffff9e6803e9ad00 x1874668790463744/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828357.314647:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828357.314658:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828357.314680:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790463744/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828357.314684:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790464512/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828357.314736:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828357.314744:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828357.314750:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790463744:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828357.314777:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790463872:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828357.314778:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828357.314794:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828357.314799:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828357.314812:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4994 00000800:00000100:2.0:1787828357.314818:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828357.314830:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828357.314842:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828357/real 1787828357] req@ffff9e6803e98b40 x1874668790463872/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828357.314846:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828357.314869:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828357.314877:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790463872/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828357.314899:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828357.314901:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790464640/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828357.314910:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828357.314924:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790463872:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828357.314947:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790464000:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828357.314975:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828357.314987:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828357/real 1787828357] req@ffff9e6803e9bc00 x1874668790464000/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828357.315000:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790464000/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828357.315011:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828357.315014:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828357.315018:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790464000:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828357.315025:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790464128:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828357.315044:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828357.315055:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828357/real 1787828357] req@ffff9e6803e9a580 x1874668790464128/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828357.315068:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790464128/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828357.315078:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828357.315081:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828357.315085:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790464128:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828357.315107:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790464256:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828357.315156:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828357.315176:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828357/real 1787828357] req@ffff9e6803e9b480 x1874668790464256/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828357.315191:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790464256/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828357.315203:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828357.315207:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828357.315211:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790464256:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828357.315218:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790464384:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828357.315244:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828357.315255:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828357/real 1787828357] req@ffff9e6803e99680 x1874668790464384/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828357.315269:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790464384/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828357.315278:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828357.315281:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828357.315285:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790464384:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828357.315292:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790464512:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828357.315310:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828357.315322:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828357/real 1787828357] req@ffff9e6803e98780 x1874668790464512/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828357.315341:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790464512/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828357.315357:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828357.315363:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828357.315369:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790464512:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828357.315379:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790464640:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828357.315419:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828357.315442:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828357/real 1787828357] req@ffff9e6803e9a1c0 x1874668790464640/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828357.315460:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790464640/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828357.315471:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828357.315475:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828357.315479:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790464640:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828357.825844:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828358.017948:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828358.849876:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828359.041947:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828359.873779:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828360.065850:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828360.897866:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828361.089986:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828361.921764:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828362.113843:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828362.433798:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828362.433808:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828362.433813:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828362.433822:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 4999 00000100:00080000:0.0:1787828362.433833:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828362.433840:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828362.433880:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790464768/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828362.433910:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828362.433915:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828362.433918:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828362.433923:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4999 00000100:00080000:0.0:1787828362.433928:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828362.433934:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828362.433946:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828362.433966:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790464896/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828362.433974:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790464768:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828362.434013:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828362.434020:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828362.434023:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828362.434029:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4999 00000100:00080000:0.0:1787828362.434034:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828362.434040:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828362.434047:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828362.434066:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790465024/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828362.434080:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828362.434133:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828362.434139:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828362.434143:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828362.434150:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4999 00000100:00080000:0.0:1787828362.434157:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828362.434164:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828362.434173:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828362.434173:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828362/real 1787828362] req@ffff9e6803e98f00 x1874668790464768/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828362.434216:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790465152/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828362.434223:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790464768/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828362.434246:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828362.434254:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828362.434256:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828362.434257:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828362.434274:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 4999 00000100:00080000:2.0:1787828362.434274:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787828362.434286:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828362.434290:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790464768:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828362.434296:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828362.434305:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828362.434320:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790465280/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828362.434329:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790464896:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828362.434337:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828362.434343:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828362.434345:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828362.434352:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4999 00000100:00080000:0.0:1787828362.434358:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828362.434363:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828362.434369:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828362.434375:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828362.434391:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790465408/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828362.434400:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828362/real 1787828362] req@ffff9e6803e9b840 x1874668790464896/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828362.434415:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828362.434428:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828362.434433:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828362.434444:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4999 00000100:00100000:2.0:1787828362.434444:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790464896/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828362.434453:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828362.434464:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787828362.434474:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787828362.434475:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828362.434485:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828362.434494:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790465536/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828362.434495:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790464896:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828362.434515:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790465024:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828362.434521:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828362.434527:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828362.434530:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828362.434537:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4999 00000100:00080000:0.0:1787828362.434543:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828362.434550:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828362.434556:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828362.434591:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828362.434599:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828362/real 1787828362] req@ffff9e6803e9a940 x1874668790465024/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828362.434606:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790465664/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828362.434626:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828362.434626:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790465024/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828362.434636:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828362.434641:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828362.434650:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828362.434651:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 4999 00000100:00080000:0.0:1787828362.434657:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828362.434660:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828362.434669:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828362.434670:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828362.434678:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790465024:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828362.434684:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790465792/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828362.434737:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790465152:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828362.434775:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828362.434789:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828362/real 1787828362] req@ffff9e6803e99e00 x1874668790465152/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828362.434800:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790465152/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828362.434808:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828362.434811:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828362.434814:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790465152:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828362.434831:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790465280:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828362.434858:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828362.434874:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828362/real 1787828362] req@ffff9e6803e9a1c0 x1874668790465280/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828362.434891:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790465280/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828362.434904:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828362.434909:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828362.434913:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790465280:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828362.434919:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790465408:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828362.434940:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828362.434951:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828362/real 1787828362] req@ffff9e6803e98780 x1874668790465408/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828362.434962:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790465408/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828362.434989:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828362.434992:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828362.434996:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790465408:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828362.435002:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790465536:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828362.435020:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828362.435031:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828362/real 1787828362] req@ffff9e6803e99680 x1874668790465536/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828362.435043:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790465536/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828362.435051:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828362.435054:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828362.435057:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790465536:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828362.435062:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790465664:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828362.435078:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828362.435087:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828362/real 1787828362] req@ffff9e6803e9b480 x1874668790465664/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828362.435146:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790465664/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828362.435160:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828362.435163:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828362.435166:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790465664:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828362.435171:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790465792:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828362.435209:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828362.435223:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828362/real 1787828362] req@ffff9e6803e9a580 x1874668790465792/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828362.435235:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790465792/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828362.435243:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828362.435246:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828362.435249:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790465792:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828362.945896:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828363.137939:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828363.969867:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828364.161958:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828364.993818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828365.185947:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828366.017881:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828366.209986:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828367.041827:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828367.233934:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828367.554158:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828367.554172:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828367.554179:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828367.554193:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5004 00000100:00080000:0.0:1787828367.554208:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828367.554218:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828367.554279:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790465920/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828367.554332:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828367.554338:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828367.554341:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828367.554348:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5004 00000100:00080000:0.0:1787828367.554354:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828367.554360:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828367.554373:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828367.554395:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790466048/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828367.554412:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828367.554417:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828367.554421:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828367.554429:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790465920:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828367.554431:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5004 00000100:00080000:0.0:1787828367.554442:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828367.554451:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828367.554458:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828367.554486:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790466176/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828367.554501:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828367.554507:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828367.554510:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828367.554517:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5004 00000100:00080000:0.0:1787828367.554540:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828367.554548:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828367.554558:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828367.554578:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790466304/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828367.554593:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828367.554598:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828367.554601:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828367.554607:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5004 00000100:00080000:0.0:1787828367.554614:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828367.554621:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790466048:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828367.554625:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828367.554641:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828367.554664:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790466432/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828367.554682:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828367.554689:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790466176:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828367.554691:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828367.554734:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828367.554746:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5004 00000100:00080000:0.0:1787828367.554753:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828367.554760:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828367.554769:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828367.554791:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790466560/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828367.554796:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790466304:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787828367.554826:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828367.554831:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828367.554834:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828367.554841:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5004 00000100:00080000:0.0:1787828367.554848:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828367.554854:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828367.554861:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790466432:192.168.203.109@tcp:8:kworker.0 00000008:00100000:0.0:1787828367.554864:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828367.554891:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790466688/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828367.554908:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828367.554916:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790466560:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828367.554918:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828367.554925:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828367.554939:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5004 00000100:00080000:0.0:1787828367.554947:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828367.554954:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828367.554963:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828367.554984:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790466816/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828367.554989:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790466688:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828367.555019:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828367.555025:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828367.555028:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828367.555034:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5004 00000100:00080000:0.0:1787828367.555041:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828367.555048:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828367.555074:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790466816:192.168.203.109@tcp:8:kworker.0 00000008:00100000:0.0:1787828367.555083:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828367.555111:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790466944/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828367.555138:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790466944:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828368.065916:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828368.257963:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828369.089848:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828369.281933:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828370.113889:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828370.305950:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787828370.626325:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:0.0:1787828370.626402:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787828370.626427:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828370.626505:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:0.0:1787828370.626551:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787828370.626554:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828367/real 1787828370] req@ffff9e6803e9a1c0 x1874668790466944/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787828370.626574:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828370.626596:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787828370.626620:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790466944/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787828370.626630:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828370.626651:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00080000:2.0:1787828370.626654:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828370.626670:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000800:00000100:0.0:1787828370.626672:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787828370.626683:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790466944:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828370.626751:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828367/real 1787828370] req@ffff9e6803e99680 x1874668790466688/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787828370.626763:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787828370.626784:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790466688/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828370.626799:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828370.626806:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828370.626812:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790466688:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828370.626823:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828367/real 1787828370] req@ffff9e6803e9a580 x1874668790466432/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828370.626843:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790466432/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828370.626854:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828370.626859:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828370.626864:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790466432:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828370.626873:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828367/real 1787828370] req@ffff9e6803e9b480 x1874668790466560/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828370.626888:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790466560/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828370.626901:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828370.626906:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828370.626919:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790466560:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828370.626929:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828367/real 1787828370] req@ffff9e6803e98b40 x1874668790466048/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828370.626946:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790466048/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828370.626959:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828370.626964:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828370.626969:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790466048:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828370.626978:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828367/real 1787828370] req@ffff9e6803e9ad00 x1874668790466176/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828370.626992:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790466176/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828370.627004:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828370.627009:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828370.627014:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790466176:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828370.627022:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828367/real 1787828370] req@ffff9e6803e992c0 x1874668790466304/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828370.627055:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790466304/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828370.627067:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828370.627071:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828370.627076:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790466304:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828370.627085:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828367/real 1787828370] req@ffff9e6803e9bc00 x1874668790465920/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828370.627102:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790465920/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828370.627113:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828370.627116:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828370.627119:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790465920:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787828370.627164:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828367/real 1787828370] req@ffff9e6803e98780 x1874668790466816/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828370.627184:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790466816/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828370.627197:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828370.627203:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828370.627207:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790466816:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828371.137864:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828371.329956:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828372.161847:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828372.353967:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828372.674330:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828372.674338:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828372.674342:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828372.674355:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5009 00000100:00080000:0.0:1787828372.674367:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828372.674375:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828372.674412:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790467072/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828372.674445:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828372.674450:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828372.674453:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828372.674457:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5009 00000100:00080000:0.0:1787828372.674462:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828372.674466:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828372.674476:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828372.674490:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790467200/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828372.674501:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828372.674505:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828372.674507:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828372.674511:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5009 00000100:00080000:0.0:1787828372.674514:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828372.674518:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828372.674524:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828372.674536:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790467328/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828372.674549:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790467072:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828372.674551:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828372.674560:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828372.674563:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828372.674568:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5009 00000100:00080000:0.0:1787828372.674572:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828372.674576:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828372.674582:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828372.674596:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790467456/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828372.674607:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828372.674611:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828372.674613:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828372.674617:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5009 00000100:00080000:0.0:1787828372.674622:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828372.674623:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828372.674630:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828372.674637:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828372.674651:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790467584/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828372.674661:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828372.674665:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828372.674670:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828372.674670:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828372/real 1787828372] req@ffff9e6803e99e00 x1874668790467072/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828372.674679:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5009 00000100:00080000:0.0:1787828372.674687:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828372.674722:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828372.674728:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828372.674734:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790467072/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828372.674750:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828372.674758:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790467712/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828372.674758:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828372.674769:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790467072:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828372.674776:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828372.674781:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828372.674782:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790467200:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828372.674785:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828372.674794:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5009 00000100:00080000:0.0:1787828372.674798:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828372.674802:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828372.674809:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828372.674820:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828372.674824:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790467840/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828372.674836:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828372.674840:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828372/real 1787828372] req@ffff9e6803e9a940 x1874668790467200/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828372.674841:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828372.674845:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828372.674854:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5009 00000100:00080000:0.0:1787828372.674861:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828372.674869:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790467200/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828372.674870:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828372.674876:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828372.674889:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828372.674890:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790467968/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828372.674898:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828372.674908:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790467200:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828372.674910:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828372.674917:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828372.674920:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828372.674926:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5009 00000100:00080000:0.0:1787828372.674930:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828372.674934:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828372.674940:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828372.674942:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790467328:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828372.674955:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790468096/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828372.674971:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828372.674988:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828372/real 1787828372] req@ffff9e6803e9b840 x1874668790467328/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828372.675002:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790467328/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828372.675010:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828372.675014:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828372.675030:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790467328:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828372.675038:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790467456:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828372.675061:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828372.675073:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828372/real 1787828372] req@ffff9e6803e98f00 x1874668790467456/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828372.675086:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790467456/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828372.675095:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828372.675099:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828372.675103:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790467456:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828372.675110:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790467584:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828372.675130:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828372.675171:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828372/real 1787828372] req@ffff9e6803e98780 x1874668790467584/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828372.675184:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790467584/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828372.675193:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828372.675196:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828372.675200:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790467584:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828372.675206:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790467712:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828372.675228:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828372.675240:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828372/real 1787828372] req@ffff9e6803e9bc00 x1874668790467712/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828372.675252:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790467712/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828372.675267:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828372.675270:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828372.675273:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790467712:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828372.675279:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790467840:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828372.675298:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828372.675309:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828372/real 1787828372] req@ffff9e6803e992c0 x1874668790467840/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828372.675321:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790467840/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828372.675337:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828372.675340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828372.675343:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790467840:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828372.675347:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790467968:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828372.675363:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828372.675372:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828372/real 1787828372] req@ffff9e6803e9ad00 x1874668790467968/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828372.675379:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790467968/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828372.675385:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828372.675387:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828372.675389:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790467968:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828372.675409:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790468096:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828372.675421:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828372.675428:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828372/real 1787828372] req@ffff9e6803e98b40 x1874668790468096/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828372.675435:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790468096/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828372.675440:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828372.675442:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828372.675445:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790468096:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828373.185807:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828373.378000:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828374.209851:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828374.401895:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828375.233829:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828375.425897:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828376.257832:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828376.450082:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828377.281886:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828377.473968:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828377.793899:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828377.793917:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828377.793924:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828377.793940:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5014 00000100:00080000:0.0:1787828377.793957:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828377.793968:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828377.794068:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790468224/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828377.794119:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828377.794124:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828377.794127:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828377.794132:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5014 00000100:00080000:0.0:1787828377.794137:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828377.794143:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828377.794156:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828377.794172:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790468352/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828377.794185:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828377.794191:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828377.794193:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828377.794198:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5014 00000100:00080000:0.0:1787828377.794203:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828377.794219:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828377.794226:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828377.794234:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790468224:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787828377.794244:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790468480/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828377.794267:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828377.794272:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828377.794285:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828377.794291:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5014 00000100:00080000:0.0:1787828377.794295:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828377.794301:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828377.794308:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828377.794327:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790468608/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828377.794338:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828377.794343:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828377.794345:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828377.794349:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5014 00000100:00080000:0.0:1787828377.794364:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828377.794369:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787828377.794384:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787828377.794389:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828377.794425:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790468736/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828377.794477:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828377/real 1787828377] req@ffff9e6803e9b480 x1874668790468224/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828377.794482:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828377.794505:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828377.794516:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828377.794525:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5014 00000100:00080000:0.0:1787828377.794531:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828377.794536:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790468224/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828377.794546:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828377.794565:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828377.794570:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828377.794579:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828377.794591:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790468224:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787828377.794595:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790468864/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828377.794607:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790468352:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828377.794644:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828377.794650:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828377.794656:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828377.794662:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828377.794666:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5014 00000100:00080000:0.0:1787828377.794679:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828377.794687:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828377.794693:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828377/real 1787828377] req@ffff9e6803e9a580 x1874668790468352/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828377.794734:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828377.794760:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790468992/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828377.794761:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790468352/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828377.794864:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828377.794885:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828377.794896:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787828377.794897:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828377.794912:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828377.794920:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790468352:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828377.794930:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5014 00000100:00080000:0.0:1787828377.794942:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828377.794949:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828377.794957:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828377.794976:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790468480:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828377.794978:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790469120/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828377.795038:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828377.795043:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828377.795045:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828377.795053:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5014 00000100:00080000:0.0:1787828377.795060:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828377.795069:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828377.795077:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828377.795088:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828377.795098:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828377/real 1787828377] req@ffff9e6803e99680 x1874668790468480/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828377.795110:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790469248/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828377.795160:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790468480/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828377.795177:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828377.795183:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828377.795188:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790468480:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828377.795216:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790468608:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828377.795241:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828377.795252:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828377/real 1787828377] req@ffff9e6803e9a1c0 x1874668790468608/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828377.795264:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790468608/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828377.795273:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828377.795293:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828377.795297:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790468608:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828377.795305:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790468736:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828377.795327:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828377.795339:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828377/real 1787828377] req@ffff9e6803e98b40 x1874668790468736/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828377.795352:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790468736/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828377.795363:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828377.795367:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828377.795370:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790468736:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828377.795378:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790468864:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828377.795397:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828377.795408:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828377/real 1787828377] req@ffff9e6803e9ad00 x1874668790468864/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828377.795421:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790468864/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828377.795443:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828377.795446:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828377.795450:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790468864:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828377.795474:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790468992:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828377.795493:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828377.795503:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828377/real 1787828377] req@ffff9e6803e992c0 x1874668790468992/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828377.795515:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790468992/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828377.795522:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828377.795525:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828377.795529:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790468992:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828377.795550:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790469120:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828377.795568:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828377.795578:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828377/real 1787828377] req@ffff9e6803e9bc00 x1874668790469120/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828377.795590:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790469120/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828377.795599:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828377.795601:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828377.795605:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790469120:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828377.795611:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790469248:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828377.795627:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828377.795636:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828377/real 1787828377] req@ffff9e6803e98780 x1874668790469248/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828377.795648:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790469248/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828377.795656:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828377.795659:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828377.795662:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790469248:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828378.305829:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828378.498003:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828379.329832:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828379.521881:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828380.353790:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828380.546049:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828381.377824:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828381.570183:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828382.401790:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828382.593859:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828382.913881:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828382.913897:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828382.913906:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828382.913921:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5019 00000100:00080000:3.0:1787828382.913940:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828382.913969:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828382.914034:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790469376/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828382.914072:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828382.914076:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828382.914079:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828382.914084:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5019 00000100:00080000:3.0:1787828382.914088:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828382.914093:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828382.914106:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828382.914121:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790469504/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828382.914136:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828382.914146:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828382.914149:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790469376:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828382.914151:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828382.914179:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5019 00000100:00080000:3.0:1787828382.914182:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828382.914186:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828382.914193:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828382.914213:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790469632/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828382.914224:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828382.914228:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828382.914248:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828382.914256:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5019 00000100:00080000:3.0:1787828382.914262:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828382.914270:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828382.914271:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828382.914290:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828382.914313:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790469760/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828382.914326:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828382.914331:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828382.914333:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828382.914341:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5019 00000100:00100000:2.0:1787828382.914344:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828382/real 1787828382] req@ffff9e680a2d6d00 x1874668790469376/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828382.914351:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828382.914365:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828382.914380:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828382.914390:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790469376/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828382.914409:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790469888/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828382.914422:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828382.914435:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828382.914439:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828382.914442:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828382.914454:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828382.914458:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790469376:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828382.914470:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5019 00000100:00080000:3.0:1787828382.914482:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828382.914492:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828382.914502:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828382.914518:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790469504:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828382.914533:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790470016/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828382.914563:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828382.914567:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828382.914570:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828382.914576:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5019 00000800:00000100:2.0:1787828382.914583:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828382.914591:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828382.914619:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828382.914627:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828382/real 1787828382] req@ffff9e680a2d4780 x1874668790469504/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828382.914631:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828382.914659:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790470144/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828382.914661:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790469504/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828382.914726:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828382.914734:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828382.914738:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828382.914749:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828382.914754:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828382.914758:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790469504:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828382.914767:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790469632:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828382.914802:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828382.914842:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5019 00000100:00100000:2.0:1787828382.914843:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828382/real 1787828382] req@ffff9e680a2d5e00 x1874668790469632/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828382.914855:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828382.914876:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828382.914879:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790469632/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828382.914890:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828382.914911:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828382.914917:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828382.914926:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790470272/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828382.914937:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790469632:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828382.914989:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790469760:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787828382.914990:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828382.915017:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828382.915021:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828382.915028:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5019 00000100:00080000:3.0:1787828382.915040:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828382.915051:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787828382.915053:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787828382.915074:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828382.915085:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828382/real 1787828382] req@ffff9e680a2d7480 x1874668790469760/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828382.915097:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790470400/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828382.915121:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790469760/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828382.915136:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828382.915141:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828382.915149:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790469760:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828382.915158:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790469888:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828382.915180:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828382.915198:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828382/real 1787828382] req@ffff9e680a2d7c00 x1874668790469888/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828382.915208:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790469888/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828382.915216:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828382.915219:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828382.915222:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790469888:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828382.915272:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790470016:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828382.915293:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828382.915304:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828382/real 1787828382] req@ffff9e680a2d6d00 x1874668790470016/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828382.915316:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790470016/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828382.915326:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828382.915329:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828382.915333:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790470016:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828382.915339:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790470144:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828382.915356:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828382.915372:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828382/real 1787828382] req@ffff9e680a2d4f00 x1874668790470144/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828382.915392:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790470144/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828382.915406:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828382.915412:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828382.915418:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790470144:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828382.915430:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790470272:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828382.915463:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828382.915478:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828382/real 1787828382] req@ffff9e680a2d52c0 x1874668790470272/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828382.915491:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790470272/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828382.915500:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828382.915504:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828382.915507:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790470272:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828382.915513:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790470400:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828382.915532:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828382.915563:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828382/real 1787828382] req@ffff9e680a2d4b40 x1874668790470400/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828382.915574:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790470400/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828407 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828382.915581:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828382.915584:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828382.915587:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790470400:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828383.425779:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828383.618037:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828384.450136:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828384.641966:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828385.473894:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828385.665993:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828386.497764:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828386.689855:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828387.521857:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828387.714015:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828388.033828:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828388.033837:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828388.033841:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828388.033851:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5024 00000100:00080000:2.0:1787828388.033861:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828388.033869:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828388.033908:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790470528/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828388.033954:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828388.033958:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828388.033970:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828388.033974:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5024 00000100:00080000:2.0:1787828388.033977:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828388.033980:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828388.033990:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828388.034002:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790470656/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828388.034012:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828388.034016:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828388.034018:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828388.034021:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5024 00000100:00080000:2.0:1787828388.034025:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828388.034028:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790470528:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828388.034031:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828388.034041:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828388.034052:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790470784/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828388.034061:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828388.034064:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828388.034066:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828388.034070:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5024 00000100:00080000:2.0:1787828388.034073:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828388.034077:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828388.034082:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828388.034090:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790470912/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828388.034098:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787828388.034103:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828388.034105:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828388.034109:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828388.034114:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5024 00000100:00080000:2.0:1787828388.034118:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828388.034122:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828388.034126:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828388.034137:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790471040/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828388.034145:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828388.034151:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828388/real 1787828388] req@ffff9e6804420f00 x1874668790470528/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828388.034152:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828388.034157:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828388.034166:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5024 00000100:00080000:2.0:1787828388.034175:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828388.034179:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828388.034184:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828388.034185:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790470528/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828388.034197:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790471168/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828388.034200:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828388.034212:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787828388.034216:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828388.034221:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790470528:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828388.034222:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828388.034228:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828388.034233:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5024 00000100:00100000:0.0:1787828388.034237:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790470656:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828388.034239:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828388.034248:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828388.034252:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828388.034264:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790471296/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828388.034272:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787828388.034275:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828388.034278:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828388.034281:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828388.034287:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5024 00000100:00080000:2.0:1787828388.034292:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828388.034296:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828388/real 1787828388] req@ffff9e6804421e00 x1874668790470656/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828388.034300:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828388.034309:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828388.034322:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790470656/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828388.034325:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790471424/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828388.034352:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828388.034354:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828388.034360:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828388.034362:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787828388.034364:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828388.034371:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790470656:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828388.034372:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5024 00000100:00080000:2.0:1787828388.034380:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828388.034386:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828388.034389:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828388.034398:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790471552/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828388.034409:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790470784:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828388.034440:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828388.034456:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828388/real 1787828388] req@ffff9e6804422940 x1874668790470784/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828388.034474:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790470784/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828388.034485:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828388.034489:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828388.034492:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790470784:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828388.034498:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790470912:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828388.034519:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828388.034534:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828388/real 1787828388] req@ffff9e6804422d00 x1874668790470912/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828388.034551:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790470912/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828388.034558:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828388.034560:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828388.034564:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790470912:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828388.034571:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790471040:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828388.034596:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828388.034611:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828388/real 1787828388] req@ffff9e6804420780 x1874668790471040/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828388.034628:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790471040/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828388.034639:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828388.034642:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828388.034645:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790471040:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828388.034650:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790471168:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828388.034670:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828388.034680:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828388/real 1787828388] req@ffff9e6804421680 x1874668790471168/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828388.034689:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790471168/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828388.034746:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828388.034750:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828388.034755:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790471168:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828388.034763:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790471296:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828388.034792:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828388.034813:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828388/real 1787828388] req@ffff9e6804422580 x1874668790471296/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828388.034827:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790471296/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828388.034835:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828388.034837:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828388.034840:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790471296:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828388.034845:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790471424:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828388.034865:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828388.034874:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828388/real 1787828388] req@ffff9e6804420b40 x1874668790471424/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828388.034883:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790471424/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828388.034889:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828388.034892:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828388.034895:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790471424:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828388.034931:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790471552:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828388.034951:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828388.034961:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828388/real 1787828388] req@ffff9e6804423840 x1874668790471552/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828388.034980:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790471552/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828388.034987:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828388.034990:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828388.034992:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790471552:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828388.545962:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828388.737867:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828389.569844:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828389.761992:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828390.594358:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828390.786586:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828391.617932:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828391.809855:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828392.642046:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828392.834185:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828393.155211:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828393.155242:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828393.155251:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828393.155270:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5029 00000100:00080000:1.0:1787828393.155292:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828393.155306:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828393.155386:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790471680/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828393.155449:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828393.155456:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828393.155460:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828393.155468:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5029 00000100:00080000:1.0:1787828393.155476:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828393.155486:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828393.155507:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828393.155537:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790471680:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828393.155537:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790471808/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828393.155588:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828393.155593:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828393.155595:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828393.155602:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5029 00000100:00080000:1.0:1787828393.155608:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828393.155613:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828393.155619:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828393.155634:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790471936/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828393.155645:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828393.155649:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828393.155663:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787828393.155664:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828393.155676:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5029 00000100:00080000:1.0:1787828393.155684:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828393.155768:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828393.155779:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828393.155817:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790472064/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828393.155818:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828393/real 1787828393] req@ffff9e6831aaad00 x1874668790471680/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828393.155926:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828393.155934:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828393.155938:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828393.155950:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5029 00000100:00100000:0.0:1787828393.155952:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790471680/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828393.155965:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828393.155987:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787828393.155995:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828393.156014:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:1.0:1787828393.156016:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828393.156024:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790471680:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828393.156043:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790472192/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828393.156058:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828393.156063:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787828393.156082:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790471808:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828393.156089:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828393.156097:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5029 00000100:00080000:1.0:1787828393.156103:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828393.156111:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828393.156120:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828393.156139:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790472320/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828393.156152:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828393.156182:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828393.156188:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828393/real 1787828393] req@ffff9e6831aabc00 x1874668790471808/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828393.156195:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828393.156208:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828393.156228:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5029 00000100:00080000:1.0:1787828393.156241:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828393.156250:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828393.156258:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828393.156263:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790471808/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828393.156281:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790472448/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828393.156309:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828393.156323:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828393.156337:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790471808:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828393.156373:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790471936:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828393.156382:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828393.156388:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828393.156392:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828393.156402:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5029 00000800:00000100:0.0:1787828393.156435:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828393.156445:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828393.156460:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828393.156472:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828393.156473:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828393/real 1787828393] req@ffff9e6831aab480 x1874668790471936/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828393.156507:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790471936/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828393.156516:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790472576/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828393.156538:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828393.156564:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828393.156586:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790471936:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828393.156594:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828393.156600:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828393.156604:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828393.156612:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790472064:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787828393.156617:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5029 00000100:00080000:1.0:1787828393.156630:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828393.156637:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828393.156644:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828393.156660:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790472704/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828393.156685:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828393.156792:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828393/real 1787828393] req@ffff9e6831aa9a40 x1874668790472064/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828393.156819:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790472064/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828393.156832:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828393.156836:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828393.156841:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790472064:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828393.156850:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790472192:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828393.156906:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828393.156925:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828393/real 1787828393] req@ffff9e6831aa8000 x1874668790472192/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828393.156940:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790472192/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828393.156951:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828393.156955:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828393.156959:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790472192:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828393.156986:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790472320:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828393.157013:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828393.157026:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828393/real 1787828393] req@ffff9e6831aaad00 x1874668790472320/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828393.157040:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790472320/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828393.157061:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828393.157065:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828393.157081:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790472320:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828393.157089:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790472448:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828393.157108:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828393.157119:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828393/real 1787828393] req@ffff9e6831aaa1c0 x1874668790472448/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828393.157133:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790472448/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828393.157143:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828393.157147:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828393.157151:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790472448:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828393.157158:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790472576:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828393.157176:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828393.157187:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828393/real 1787828393] req@ffff9e6831aa8780 x1874668790472576/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828393.157202:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790472576/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828393.157219:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828393.157225:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828393.157233:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790472576:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828393.157246:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790472704:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828393.157287:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828393.157309:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828393/real 1787828393] req@ffff9e6831aaa940 x1874668790472704/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828393.157325:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790472704/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828393.157336:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828393.157340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828393.157344:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790472704:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828393.665797:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828393.857896:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828394.689861:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828394.882057:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828395.713934:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828395.906064:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828396.738437:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828396.929903:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828397.761904:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828397.954013:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828398.273915:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828398.273928:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828398.273935:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828398.273947:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5034 00000100:00080000:3.0:1787828398.273961:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828398.273971:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828398.274020:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790472832/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828398.274057:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828398.274062:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828398.274065:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828398.274070:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5034 00000100:00080000:3.0:1787828398.274074:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828398.274078:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828398.274092:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828398.274109:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790472960/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828398.274124:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828398.274129:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828398.274133:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828398.274139:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5034 00000100:00080000:3.0:1787828398.274143:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828398.274147:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828398.274155:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790472832:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787828398.274157:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828398.274178:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790473088/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828398.274200:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828398.274204:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828398.274207:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828398.274212:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5034 00000100:00080000:3.0:1787828398.274218:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828398.274223:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828398.274229:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828398.274246:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790473216/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828398.274257:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828398.274272:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828398.274277:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828398.274279:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828398.274287:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5034 00000100:00080000:3.0:1787828398.274292:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828398.274298:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828398.274305:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828398.274321:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828398/real 1787828398] req@ffff9e680a2d61c0 x1874668790472832/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828398.274324:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790473344/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828398.274388:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828398.274392:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828398.274394:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828398.274401:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5034 00000100:00080000:3.0:1787828398.274409:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828398.274416:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828398.274422:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790472832/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828398.274425:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828398.274457:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790473472/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828398.274459:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828398.274477:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828398.274483:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828398.274491:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790472832:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828398.274494:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828398.274503:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828398.274514:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5034 00000100:00080000:3.0:1787828398.274518:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828398.274523:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828398.274528:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828398.274538:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790472960:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828398.274548:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790473600/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828398.274567:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828398.274571:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828398.274575:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787828398.274587:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828398.274588:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5034 00000100:00080000:3.0:1787828398.274607:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828398.274620:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828398/real 1787828398] req@ffff9e680a2d70c0 x1874668790472960/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828398.274622:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828398.274645:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828398.274667:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790472960/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828398.274682:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828398.274687:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828398.274736:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790472960:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828398.274744:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790473728/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828398.274749:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790473088:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828398.274787:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787828398.274795:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828398.274798:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828398.274804:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828398.274815:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5034 00000100:00080000:3.0:1787828398.274824:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828398.274825:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828398/real 1787828398] req@ffff9e680a2d4000 x1874668790473088/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828398.274838:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828398.274869:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828398.274885:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790473088/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828398.274896:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790473856/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828398.274940:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828398.274953:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828398.274961:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790473088:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828398.274971:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790473216:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828398.275000:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828398.275013:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828398/real 1787828398] req@ffff9e680a2d5a40 x1874668790473216/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828398.275027:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790473216/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828398.275038:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828398.275041:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828398.275045:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790473216:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828398.275052:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790473344:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828398.275071:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828398.275082:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828398/real 1787828398] req@ffff9e680a2d43c0 x1874668790473344/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828398.275095:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790473344/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828398.275104:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828398.275108:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828398.275112:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790473344:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828398.275119:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790473472:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828398.275137:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828398.275148:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828398/real 1787828398] req@ffff9e680a2d6580 x1874668790473472/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828398.275161:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790473472/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828398.275172:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828398.275176:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828398.275180:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790473472:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828398.275205:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790473600:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828398.275225:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828398.275237:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828398/real 1787828398] req@ffff9e680a2d61c0 x1874668790473600/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828398.275250:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790473600/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828398.275260:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828398.275263:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828398.275268:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790473600:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828398.275275:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790473728:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828398.275293:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828398.275304:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828398/real 1787828398] req@ffff9e680a2d4b40 x1874668790473728/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828398.275318:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790473728/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828398.275327:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828398.275330:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828398.275334:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790473728:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828398.275340:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790473856:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828398.275357:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828398.275369:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828398/real 1787828398] req@ffff9e680a2d52c0 x1874668790473856/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828398.275382:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790473856/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828398.275392:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828398.275395:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828398.275399:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790473856:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828398.785914:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828398.977908:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828399.810253:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828400.001913:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828400.833912:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828401.025862:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828401.857927:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828402.049882:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828402.882530:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828403.073850:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828403.393874:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828403.393883:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828403.393887:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828403.393907:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5040 00000100:00080000:3.0:1787828403.393917:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828403.393923:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828403.393959:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790473984/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828403.393994:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828403.393998:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828403.394001:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828403.394006:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5040 00000100:00080000:3.0:1787828403.394010:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828403.394015:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828403.394033:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828403.394040:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790473984:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828403.394053:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790474112/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828403.394083:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828403.394088:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828403.394090:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828403.394093:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5040 00000100:00080000:3.0:1787828403.394098:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828403.394102:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828403.394108:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828403.394119:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828403.394125:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790474240/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828403.394138:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828403.394142:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828403.394144:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828403.394149:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5040 00000100:00080000:3.0:1787828403.394154:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828403.394159:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828403/real 1787828403] req@ffff9e680a2d4f00 x1874668790473984/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828403.394164:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828403.394176:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828403.394183:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790473984/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828403.394194:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790474368/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828403.394195:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828403.394201:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828403.394220:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790473984:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828403.394221:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828403.394228:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828403.394230:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828403.394235:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5040 00000100:00080000:3.0:1787828403.394239:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828403.394243:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828403.394248:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790474112:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787828403.394248:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828403.394262:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790474496/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828403.394273:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828403.394276:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828403.394278:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828403.394280:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5040 00000100:00080000:3.0:1787828403.394284:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828403.394286:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828403.394290:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828403.394295:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828403.394305:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790474624/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828403.394313:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828403/real 1787828403] req@ffff9e680a2d6d00 x1874668790474112/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828403.394321:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828403.394325:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828403.394345:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828403.394349:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790474112/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828403.394355:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5040 00000100:00080000:3.0:1787828403.394365:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828403.394374:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787828403.394376:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828403.394411:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787828403.394412:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828403.394420:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790474112:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828403.394431:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790474752/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828403.394432:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790474240:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828403.394452:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828403.394455:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828403.394474:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828403.394479:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5040 00000100:00080000:3.0:1787828403.394485:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828403.394490:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828403.394495:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828403.394502:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828403.394511:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828403/real 1787828403] req@ffff9e680a2d7c00 x1874668790474240/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828403.394517:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790474880/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828403.394558:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790474240/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828403.394563:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828403.394568:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828403.394569:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828403.394573:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828403.394576:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828403.394580:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5040 00000100:00100000:0.0:1787828403.394582:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790474240:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828403.394585:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828403.394590:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790474368:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787828403.394592:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828403.394599:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828403.394611:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790475008/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828403.394621:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828403.394655:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828403/real 1787828403] req@ffff9e680a2d7480 x1874668790474368/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828403.394672:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790474368/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828403.394684:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828403.394687:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828403.394699:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790474368:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828403.394738:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790474496:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828403.394766:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828403.394780:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828403/real 1787828403] req@ffff9e680a2d5e00 x1874668790474496/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828403.394875:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790474496/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828403.394882:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828403.394884:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828403.394887:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790474496:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828403.394891:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790474624:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828403.394926:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828403.394937:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828403/real 1787828403] req@ffff9e680a2d4780 x1874668790474624/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828403.394948:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790474624/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828403.394956:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828403.394959:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828403.394961:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790474624:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828403.394966:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790474752:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828403.394981:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828403.394989:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828403/real 1787828403] req@ffff9e680a2d4f00 x1874668790474752/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828403.394999:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790474752/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828403.395005:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828403.395008:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828403.395010:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790474752:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828403.395015:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790474880:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828403.395028:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828403.395036:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828403/real 1787828403] req@ffff9e680a2d52c0 x1874668790474880/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828403.395046:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790474880/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828403.395053:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828403.395055:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828403.395058:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790474880:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828403.395062:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790475008:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828403.395075:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828403.395082:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828403/real 1787828403] req@ffff9e680a2d4b40 x1874668790475008/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828403.395091:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790475008/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828403.395098:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828403.395101:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828403.395104:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790475008:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828403.905840:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828404.097957:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828404.929829:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828405.121934:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828405.953972:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828406.145930:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828406.977938:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828407.170003:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828408.002231:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828408.193957:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828408.513926:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828408.513939:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828408.513946:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828408.513959:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5045 00000100:00080000:3.0:1787828408.513975:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828408.513986:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828408.514046:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790475136/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828408.514078:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828408.514082:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828408.514085:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828408.514091:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5045 00000100:00080000:3.0:1787828408.514096:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828408.514102:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828408.514116:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828408.514132:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790475136:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828408.514136:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790475264/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828408.514157:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828408.514162:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828408.514165:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828408.514170:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5045 00000100:00080000:3.0:1787828408.514175:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828408.514178:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828408.514185:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828408.514199:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790475392/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828408.514218:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828408.514225:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828408.514229:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828408.514236:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5045 00000800:00000100:0.0:1787828408.514246:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828408.514258:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828408.514275:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828408.514288:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828408.514313:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828408/real 1787828408] req@ffff9e680a2d61c0 x1874668790475136/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828408.514318:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790475520/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828408.514364:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828408.514370:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790475136/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828408.514374:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828408.514379:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828408.514389:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5045 00000100:00080000:3.0:1787828408.514402:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828408.514413:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828408.514417:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787828408.514425:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828408.514434:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790475136:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787828408.514435:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828408.514466:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790475648/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828408.514475:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790475264:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828408.514509:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828408.514517:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828408.514522:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787828408.514544:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828408.514554:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5045 00000100:00080000:3.0:1787828408.514565:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828408.514573:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828408/real 1787828408] req@ffff9e680a2d6580 x1874668790475264/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828408.514581:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828408.514597:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828408.514606:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790475264/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828408.514620:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828408.514623:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790475776/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828408.514630:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828408.514644:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790475264:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828408.514657:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828408.514665:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790475392:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828408.514666:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828408.514673:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828408.514682:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5045 00000100:00080000:3.0:1787828408.514687:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828408.514712:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828408.514720:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828408.514757:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828408.514822:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828408/real 1787828408] req@ffff9e680a2d43c0 x1874668790475392/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828408.514828:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790475904/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828408.514859:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828408.514862:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790475392/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828408.514865:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828408.514871:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828408.514884:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5045 00000100:00080000:0.0:1787828408.514886:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828408.514890:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828408.514896:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828408.514902:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828408.514906:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790475392:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787828408.514911:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828408.514918:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790475520:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828408.514927:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790476032/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828408.514940:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828408.514944:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828408.514947:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787828408.514951:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828408.514957:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5045 00000100:00080000:3.0:1787828408.514963:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828408.514976:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828408.514982:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828408.514983:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828408/real 1787828408] req@ffff9e680a2d5a40 x1874668790475520/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828408.515000:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790476160/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828408.515005:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790475520/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828408.515037:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828408.515060:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828408.515067:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790475520:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828408.515099:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790475648:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828408.515140:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828408.515152:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828408/real 1787828408] req@ffff9e680a2d4000 x1874668790475648/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828408.515168:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790475648/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828408.515178:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828408.515182:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828408.515187:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790475648:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828408.515197:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790475776:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828408.515227:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828408.515244:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828408/real 1787828408] req@ffff9e680a2d70c0 x1874668790475776/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828408.515281:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790475776/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828408.515295:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828408.515299:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828408.515303:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790475776:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828408.515312:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790475904:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828408.515345:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828408.515370:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828408/real 1787828408] req@ffff9e680a2d61c0 x1874668790475904/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828408.515387:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790475904/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828408.515424:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828408.515430:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828408.515436:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790475904:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828408.515448:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790476032:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828408.515478:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828408.515493:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828408/real 1787828408] req@ffff9e680a2d4b40 x1874668790476032/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828408.515506:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790476032/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828408.515515:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828408.515518:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828408.515522:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790476032:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828408.515529:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790476160:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828408.515546:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828408.515557:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828408/real 1787828408] req@ffff9e680a2d52c0 x1874668790476160/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828408.515572:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790476160/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828408.515586:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828408.515592:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828408.515599:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790476160:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828409.025867:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828409.218006:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828410.049899:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828410.241958:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828411.073918:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828411.265997:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828412.097903:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828412.290031:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828413.122323:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828413.313949:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828413.633808:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828413.633819:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828413.633824:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828413.633834:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5050 00000100:00080000:1.0:1787828413.633843:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828413.633849:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828413.633886:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790476288/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828413.633911:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828413.633915:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828413.633916:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828413.633920:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5050 00000100:00080000:1.0:1787828413.633923:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828413.633926:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828413.633935:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828413.633954:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790476416/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828413.633969:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828413.633972:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790476288:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828413.633978:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828413.633983:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828413.633990:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5050 00000100:00080000:1.0:1787828413.634007:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828413.634012:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828413.634017:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828413.634035:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790476544/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828413.634048:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787828413.634053:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828413.634058:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828413.634063:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828413.634071:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5050 00000100:00080000:1.0:1787828413.634077:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828413.634083:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828413.634090:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828413.634098:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828413/real 1787828413] req@ffff9e6831aa9e00 x1874668790476288/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828413.634108:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790476672/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828413.634120:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828413.634122:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790476288/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828413.634124:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828413.634130:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828413.634141:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828413.634142:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5050 00000100:00080000:1.0:1787828413.634152:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828413.634153:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828413.634164:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828413.634165:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790476288:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787828413.634176:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828413.634194:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790476800/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828413.634203:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790476416:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828413.634207:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828413.634215:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828413.634217:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828413.634221:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5050 00000100:00080000:1.0:1787828413.634225:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828413.634229:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828413.634234:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828413.634242:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828413.634247:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790476928/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828413.634257:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828413.634260:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828413/real 1787828413] req@ffff9e6831aa8b40 x1874668790476416/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828413.634262:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828413.634266:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828413.634277:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5050 00000100:00080000:1.0:1787828413.634282:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828413.634286:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790476416/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828413.634298:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828413.634309:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787828413.634313:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828413.634320:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828413.634323:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790477056/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828413.634327:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790476416:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828413.634342:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790476544:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828413.634348:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828413.634364:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828413.634368:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787828413.634374:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828413.634376:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5050 00000100:00080000:1.0:1787828413.634386:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828413.634387:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828413/real 1787828413] req@ffff9e6831aa9680 x1874668790476544/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828413.634398:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828413.634405:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790476544/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828413.634408:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787828413.634419:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828413.634423:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828413.634430:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790476544:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828413.634430:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790477184/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828413.634444:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790476672:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787828413.634459:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828413.634464:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1787828413.634468:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828413.634470:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828413.634481:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828413/real 1787828413] req@ffff9e6831aaa580 x1874668790476672/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828413.634481:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5050 00000100:00080000:1.0:1787828413.634491:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828413.634497:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790476672/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828413.634515:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787828413.634520:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787828413.634525:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787828413.634527:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828413.634533:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790476672:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828413.634545:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790477312/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828413.634552:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790476800:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828413.634576:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828413.634586:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828413/real 1787828413] req@ffff9e6831aa8f00 x1874668790476800/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828413.634595:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790476800/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828413.634603:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828413.634606:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828413.634608:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790476800:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828413.634612:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790476928:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828413.634624:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828413.634630:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828413/real 1787828413] req@ffff9e6831aa9e00 x1874668790476928/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828413.634657:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790476928/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828413.634665:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828413.634667:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828413.634670:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790476928:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828413.634674:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790477056:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828413.634687:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828413.634723:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828413/real 1787828413] req@ffff9e6831aaa940 x1874668790477056/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828413.634751:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790477056/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828413.634759:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828413.634762:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828413.634766:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790477056:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828413.634771:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790477184:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828413.634801:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828413.634811:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828413/real 1787828413] req@ffff9e6831aa8780 x1874668790477184/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828413.634820:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790477184/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828413.634827:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828413.634829:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828413.634831:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790477184:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828413.634861:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790477312:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828413.634886:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828413.634901:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828413/real 1787828413] req@ffff9e6831aaa1c0 x1874668790477312/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828413.634916:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790477312/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828413.634936:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828413.634939:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828413.634942:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790477312:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828414.145915:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828414.337904:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828415.170092:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828415.361949:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828416.193943:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828416.386054:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828417.217872:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828417.409971:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828418.241896:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828418.434281:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828418.754259:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828418.754275:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828418.754282:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828418.754298:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5055 00000100:00080000:3.0:1787828418.754315:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828418.754327:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828418.754382:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790477440/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828418.754423:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828418.754428:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828418.754431:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828418.754438:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5055 00000100:00080000:3.0:1787828418.754443:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828418.754449:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828418.754462:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828418.754481:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790477568/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828418.754496:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828418.754500:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828418.754503:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828418.754511:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5055 00000100:00100000:0.0:1787828418.754514:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790477440:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828418.754516:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828418.754526:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828418.754531:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828418.754543:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790477696/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828418.754558:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828418.754562:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828418.754564:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828418.754568:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5055 00000100:00080000:3.0:1787828418.754571:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828418.754574:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828418.754589:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828418.754599:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790477824/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828418.754607:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828418.754610:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828418.754611:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828418.754614:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5055 00000100:00080000:3.0:1787828418.754617:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828418.754620:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828418.754623:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828418.754631:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790477952/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828418.754641:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787828418.754642:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828418.754645:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828418.754649:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828418.754653:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5055 00000100:00080000:3.0:1787828418.754657:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828418.754661:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828418.754664:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828418.754672:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790478080/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828418.754680:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828418.754683:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828418.754684:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828418.754687:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5055 00000100:00080000:3.0:1787828418.754690:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828418.754777:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828418.754785:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828418.754792:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828418/real 1787828418] req@ffff9e680a2d4f00 x1874668790477440/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828418.754820:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790478208/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828418.754832:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828418.754837:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787828418.754839:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790477440/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828418.754840:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828418.754850:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5055 00000100:00080000:3.0:1787828418.754857:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828418.754865:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787828418.754869:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828418.754870:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787828418.754880:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828418.754886:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790478336/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828418.754891:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790477440:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828418.754910:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828418.754913:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828418.754915:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828418.754920:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5055 00000100:00080000:3.0:1787828418.754932:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828418.754935:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828418.754937:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828418.754951:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790477568:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828418.754951:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790478464/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828418.755008:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828418.755027:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828418/real 1787828418] req@ffff9e680a2d4780 x1874668790477568/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828418.755043:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790477568/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828418.755054:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828418.755058:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828418.755062:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790477568:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828418.755069:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790477696:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828418.755109:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828418.755123:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828418/real 1787828418] req@ffff9e680a2d5e00 x1874668790477696/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828418.755138:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790477696/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828418.755178:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828418.755182:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828418.755186:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790477696:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828418.755194:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790477824:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828418.755216:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828418.755229:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828418/real 1787828418] req@ffff9e680a2d7480 x1874668790477824/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828418.755243:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790477824/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828418.755254:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828418.755257:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828418.755261:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790477824:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828418.755269:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790477952:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828418.755290:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828418.755302:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828418/real 1787828418] req@ffff9e680a2d7c00 x1874668790477952/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828418.755316:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790477952/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828418.755326:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828418.755330:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828418.755335:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790477952:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828418.755342:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790478080:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828418.755362:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828418.755375:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828418/real 1787828418] req@ffff9e680a2d6d00 x1874668790478080/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828418.755391:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790478080/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828418.755403:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828418.755407:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828418.755412:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790478080:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828418.755420:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790478208:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828418.755446:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828418.755460:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828418/real 1787828418] req@ffff9e680a2d52c0 x1874668790478208/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828418.755475:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790478208/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828418.755485:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828418.755490:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828418.755493:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790478208:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828418.755501:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790478336:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828418.755525:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828418.755541:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828418/real 1787828418] req@ffff9e680a2d4b40 x1874668790478336/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828418.755557:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790478336/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828418.755569:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828418.755574:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828418.755578:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790478336:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828418.755611:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790478464:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828418.755636:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828418.755651:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828418/real 1787828418] req@ffff9e680a2d61c0 x1874668790478464/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828418.755668:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790478464/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828418.755679:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828418.755684:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828418.755688:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790478464:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828419.266334:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828419.457981:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828420.290457:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828420.482036:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828421.314183:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828421.505925:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828422.337866:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828422.529823:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828423.361833:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828423.553994:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828423.874358:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828423.874372:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828423.874378:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828423.874392:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5060 00000100:00080000:3.0:1787828423.874406:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828423.874417:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828423.874465:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790478592/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828423.874562:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790478592:192.168.203.109@tcp:250:kworker.0 00000800:00000100:0.0:1787828423.874630:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828423.874816:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828423.874822:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828423.874825:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828423.874830:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5060 00000100:00080000:3.0:1787828423.874835:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828423.874840:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828423.874851:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828423.874867:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790478720/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828423.874976:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828423/real 1787828423] req@ffff9e680a2d70c0 x1874668790478592/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828423.874997:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790478592/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828423.875007:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828423.875012:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828423.875015:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790478592:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787828423.875040:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790478720:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828423.875066:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828423.875078:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828423/real 1787828423] req@ffff9e680a2d4000 x1874668790478720/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828423.875090:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790478720/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828423.875098:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828423.875101:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828423.875104:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790478720:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828423.875160:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828423.875165:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828423.875167:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828423.875172:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5060 00000100:00080000:3.0:1787828423.875176:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828423.875181:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828423.875186:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828423.875200:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790478848/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828423.875211:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828423.875214:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828423.875216:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828423.875219:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5060 00000100:00080000:3.0:1787828423.875223:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828423.875226:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828423.875229:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828423.875238:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790478976/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828423.875246:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828423.875249:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828423.875251:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828423.875254:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5060 00000100:00080000:3.0:1787828423.875257:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828423.875260:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828423.875263:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828423.875271:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790479104/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828423.875278:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828423.875281:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828423.875282:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828423.875286:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5060 00000100:00080000:3.0:1787828423.875288:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828423.875292:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828423.875295:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828423.875303:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790479232/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828423.875311:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828423.875314:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828423.875315:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828423.875319:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5060 00000100:00080000:3.0:1787828423.875322:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828423.875325:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828423.875329:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828423.875335:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790479360/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828423.875342:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828423.875345:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828423.875346:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828423.875349:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5060 00000100:00080000:3.0:1787828423.875352:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828423.875354:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828423.875358:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828423.875364:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790479488/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828423.875372:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828423.875375:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828423.875376:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828423.875379:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5060 00000100:00080000:3.0:1787828423.875381:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828423.875384:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828423.875387:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828423.875394:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790479616/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828423.875463:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790478848:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828423.875489:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828423.875501:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828423/real 1787828423] req@ffff9e680a2d5a40 x1874668790478848/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828423.875513:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790478848/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828423.875528:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828423.875531:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828423.875535:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790478848:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828423.875540:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790478976:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828423.875556:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828423.875564:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828423/real 1787828423] req@ffff9e680a2d43c0 x1874668790478976/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828423.875575:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790478976/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828423.875583:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828423.875585:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828423.875588:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790478976:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828423.875593:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790479104:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828423.875607:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828423.875615:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828423/real 1787828423] req@ffff9e680a2d6580 x1874668790479104/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828423.875626:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790479104/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828423.875633:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828423.875636:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828423.875639:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790479104:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828423.875644:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790479232:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828423.875658:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828423.875692:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828423/real 1787828423] req@ffff9e680a2d4000 x1874668790479232/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828423.875798:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790479232/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828423.875807:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828423.875810:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828423.875813:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790479232:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828423.875819:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790479360:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828423.875844:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828423.875856:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828423/real 1787828423] req@ffff9e680a2d70c0 x1874668790479360/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828423.875866:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790479360/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828423.875874:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828423.875876:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828423.875879:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790479360:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828423.875884:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790479488:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828423.875923:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828423.875938:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828423/real 1787828423] req@ffff9e680a2d61c0 x1874668790479488/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828423.875953:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790479488/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828423.875978:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828423.875981:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828423.875984:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790479488:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828423.875989:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790479616:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828423.876006:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828423.876015:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828423/real 1787828423] req@ffff9e680a2d4b40 x1874668790479616/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828423.876026:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790479616/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828448 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828423.876033:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828423.876035:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828423.876038:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790479616:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828424.386804:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828424.577861:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828425.409827:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828425.601976:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828426.434421:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828426.625872:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828427.457860:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828427.649943:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828428.481831:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828428.673904:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828428.993850:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828428.993867:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828428.993875:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828428.993894:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5065 00000100:00080000:3.0:1787828428.993911:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828428.993924:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828428.993995:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790479744/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828428.994033:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828428.994038:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828428.994042:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828428.994050:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5065 00000100:00080000:3.0:1787828428.994057:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828428.994065:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828428.994081:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828428.994105:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790479872/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828428.994126:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828428.994134:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828428.994138:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828428.994147:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5065 00000100:00080000:3.0:1787828428.994156:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828428.994166:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790479744:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828428.994168:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828428.994181:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828428.994203:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790480000/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828428.994218:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828428.994223:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828428.994226:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828428.994231:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5065 00000100:00080000:3.0:1787828428.994239:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828428.994247:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828428.994257:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828428.994277:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828428.994286:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790480128/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828428.994309:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828428.994316:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828428.994321:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828428.994330:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5065 00000100:00080000:3.0:1787828428.994339:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828428.994349:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828428.994358:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828428/real 1787828428] req@ffff9e680a2d52c0 x1874668790479744/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828428.994360:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828428.994389:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790480256/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828428.994400:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790479744/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828428.994411:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828428.994426:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828428.994433:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828428.994440:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828428.994444:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5065 00000100:00080000:3.0:1787828428.994455:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828428.994456:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828428.994469:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828428.994472:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790479744:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787828428.994478:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828428.994489:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790479872:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828428.994499:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790480384/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828428.994514:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828428.994518:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828428.994521:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828428.994527:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5065 00000100:00080000:3.0:1787828428.994532:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828428.994540:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828428.994547:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828428.994556:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828428.994569:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828428/real 1787828428] req@ffff9e680a2d6d00 x1874668790479872/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828428.994576:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790480512/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828428.994620:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828428.994655:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790479872/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828428.994655:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828428.994662:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828428.994689:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5065 00000100:00080000:0.0:1787828428.994738:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828428.994742:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828428.994749:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828428.994757:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828428.994766:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790479872:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787828428.994771:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828428.994798:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790480640/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828428.994814:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828428.994816:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790480000:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828428.994821:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828428.994829:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828428.994836:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5065 00000100:00080000:3.0:1787828428.994841:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828428.994847:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828428.994856:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828428.994865:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828428.994886:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790480768/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828428.994892:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828428/real 1787828428] req@ffff9e680a2d7c00 x1874668790480000/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828428.994945:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790480000/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828428.994959:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828428.994964:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828428.994971:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790480000:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828428.994987:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790480128:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828428.995007:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828428.995018:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828428/real 1787828428] req@ffff9e680a2d7480 x1874668790480128/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828428.995032:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790480128/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828428.995042:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828428.995046:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828428.995049:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790480128:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828428.995056:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790480256:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828428.995073:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828428.995084:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828428/real 1787828428] req@ffff9e680a2d5e00 x1874668790480256/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828428.995096:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790480256/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828428.995106:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828428.995109:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828428.995112:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790480256:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828428.995119:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790480384:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828428.995142:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828428.995161:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828428/real 1787828428] req@ffff9e680a2d4780 x1874668790480384/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828428.995186:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790480384/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828428.995206:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828428.995214:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828428.995223:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790480384:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828428.995236:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790480512:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828428.995273:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828428.995288:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828428/real 1787828428] req@ffff9e680a2d4f00 x1874668790480512/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828428.995307:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790480512/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828428.995320:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828428.995325:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828428.995329:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790480512:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828428.995353:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790480640:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828428.995374:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828428.995386:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828428/real 1787828428] req@ffff9e680a2d4b40 x1874668790480640/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828428.995399:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790480640/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828428.995418:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828428.995421:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828428.995425:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790480640:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828428.995431:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790480768:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828428.995449:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828428.995459:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828428/real 1787828428] req@ffff9e680a2d61c0 x1874668790480768/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828428.995472:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790480768/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828453 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828428.995481:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828428.995484:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828428.995488:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790480768:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828429.505804:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828429.697965:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828430.530115:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828430.722032:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828431.553926:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828431.745920:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828432.577845:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828432.770049:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828433.601827:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828433.793912:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828434.113857:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828434.113872:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828434.113879:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828434.113893:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5070 00000100:00080000:0.0:1787828434.113908:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828434.113919:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828434.113967:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790480896/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828434.114000:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828434.114005:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828434.114007:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828434.114011:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5070 00000100:00080000:0.0:1787828434.114015:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828434.114021:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828434.114032:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828434.114046:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790481024/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828434.114059:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790480896:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828434.114063:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828434.114073:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828434.114075:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828434.114081:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5070 00000100:00080000:0.0:1787828434.114084:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828434.114088:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828434.114092:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828434.114106:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790481152/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828434.114116:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828434.114120:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828434.114122:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828434.114127:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5070 00000100:00080000:0.0:1787828434.114131:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828434.114134:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828434.114140:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828434.114152:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790481280/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828434.114161:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828434.114165:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828434.114167:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828434.114172:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5070 00000100:00080000:0.0:1787828434.114174:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828434.114177:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828434.114183:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828434.114203:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790481408/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828434.114209:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790481024:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828434.114237:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828434.114243:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828434.114247:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828434.114255:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5070 00000100:00080000:0.0:1787828434.114263:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828434.114271:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828434.114275:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790481152:192.168.203.109@tcp:38:kworker.0 00000008:00100000:0.0:1787828434.114284:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828434.114308:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790481536/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828434.114315:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790481280:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787828434.114333:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828434.114337:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828434.114341:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828434.114347:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5070 00000100:00080000:0.0:1787828434.114353:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828434.114357:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828434.114363:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828434.114375:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790481408:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828434.114379:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790481664/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828434.114397:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828434.114401:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828434.114403:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828434.114408:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5070 00000100:00080000:0.0:1787828434.114414:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828434.114418:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828434.114424:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828434.114435:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790481536:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828434.114444:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790481792/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828434.114457:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828434.114461:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828434.114464:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828434.114468:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5070 00000100:00080000:0.0:1787828434.114473:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828434.114478:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790481664:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828434.114481:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828434.114493:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828434.114509:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790481920/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828434.114519:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790481792:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828434.114549:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790481920:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828434.627125:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828434.817873:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828435.650487:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828435.842013:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828436.673869:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828436.865992:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787828437.186232:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:3.0:1787828437.186336:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787828437.186377:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787828437.186507:0:2386:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:3.0:1787828437.186555:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787828437.186603:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828434/real 1787828437] req@ffff9e6803e98b40 x1874668790481920/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787828437.186618:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787828437.186650:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787828437.186673:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790481920/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787828437.186686:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00080000:2.0:1787828437.186801:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828437.186819:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000800:00000100:3.0:1787828437.186821:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787828437.186837:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790481920:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828437.186865:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787828437.186877:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828434/real 1787828437] req@ffff9e6803e98780 x1874668790481408/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787828437.186897:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787828437.186934:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790481408/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828437.186951:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828437.186959:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828437.186965:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790481408:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828437.186979:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828434/real 1787828437] req@ffff9e6803e9bc00 x1874668790481536/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828437.187004:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790481536/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828437.187022:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828437.187028:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828437.187035:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790481536:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828437.187048:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828434/real 1787828437] req@ffff9e6803e9b840 x1874668790481024/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828437.187073:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790481024/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828437.187089:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828437.187096:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828437.187103:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790481024:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828437.187116:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828434/real 1787828437] req@ffff9e6803e9a940 x1874668790481152/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828437.187140:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790481152/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828437.187155:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828437.187162:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828437.187169:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790481152:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828437.187196:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828434/real 1787828437] req@ffff9e6803e99e00 x1874668790481280/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828437.187212:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790481280/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828437.187224:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828437.187229:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828437.187234:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790481280:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828437.187244:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828434/real 1787828437] req@ffff9e6803e98f00 x1874668790480896/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828437.187263:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790480896/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828437.187275:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828437.187280:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828437.187285:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790480896:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787828437.187333:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828434/real 1787828437] req@ffff9e6803e992c0 x1874668790481664/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828437.187352:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790481664/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828437.187363:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828437.187369:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828437.187374:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790481664:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828437.187384:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828434/real 1787828437] req@ffff9e6803e9ad00 x1874668790481792/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828437.187399:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790481792/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828437.187409:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828437.187414:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828437.187418:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790481792:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828437.697878:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828437.890019:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828438.722641:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828438.913954:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828439.233859:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828439.233876:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828439.233883:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828439.233901:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5075 00000100:00080000:3.0:1787828439.233917:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828439.233929:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828439.233977:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790482048/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828439.234030:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828439.234038:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828439.234042:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828439.234051:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5075 00000100:00080000:3.0:1787828439.234061:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828439.234068:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828439.234085:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828439.234116:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790482176/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828439.234131:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790482048:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828439.234154:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828439.234161:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828439.234164:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828439.234172:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5075 00000100:00080000:3.0:1787828439.234180:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828439.234190:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828439.234201:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828439.234245:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790482304/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828439.234269:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828439.234273:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828439.234281:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828439.234285:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828439.234294:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5075 00000100:00080000:3.0:1787828439.234302:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828439.234310:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828439.234322:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828439.234355:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828439/real 1787828439] req@ffff9e680a2d70c0 x1874668790482048/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828439.234373:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790482432/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828439.234414:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828439.234418:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790482048/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828439.234421:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828439.234430:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828439.234454:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828439.234456:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5075 00000100:00080000:3.0:1787828439.234474:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828439.234476:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828439.234491:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828439.234495:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790482048:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787828439.234507:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828439.234540:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790482560/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828439.234592:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790482176:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828439.234596:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828439.234612:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828439.234616:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828439.234629:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5075 00000100:00080000:3.0:1787828439.234639:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828439.234649:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828439.234660:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828439.234671:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828439.234695:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790482688/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828439.234758:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828439/real 1787828439] req@ffff9e680a2d4000 x1874668790482176/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828439.234829:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828439.234841:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828439.234846:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828439.234854:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790482176/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828439.234863:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5075 00000100:00080000:3.0:1787828439.234880:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828439.234893:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828439.234897:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787828439.234910:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:3.0:1787828439.234915:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828439.234924:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790482176:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828439.234945:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790482304:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828439.234952:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790482816/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828439.234977:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828439.234984:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828439.234989:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828439.234997:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5075 00000800:00000100:2.0:1787828439.235009:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828439.235009:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828439.235026:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828439.235038:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828439.235049:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828439/real 1787828439] req@ffff9e680a2d6580 x1874668790482304/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828439.235069:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790482944/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828439.235094:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790482304/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828439.235095:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828439.235115:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828439.235124:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828439.235139:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828439.235148:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828439.235155:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790482304:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828439.235170:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790482432:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828439.235214:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828439.235239:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828439/real 1787828439] req@ffff9e680a2d43c0 x1874668790482432/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828439.235264:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790482432/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828439.235292:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828439.235299:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828439.235307:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790482432:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828439.235352:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790482560:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787828439.235357:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5075 00000100:00080000:3.0:1787828439.235375:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828439.235386:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828439.235398:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828439.235424:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828439.235431:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790483072/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828439.235461:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828439/real 1787828439] req@ffff9e680a2d5a40 x1874668790482560/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828439.235498:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790482560/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828439.235517:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828439.235525:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828439.235533:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790482560:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828439.235546:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790482688:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828439.235617:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828439.235640:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828439/real 1787828439] req@ffff9e680a2d70c0 x1874668790482688/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828439.235656:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790482688/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828439.235680:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828439.235686:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828439.235732:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790482688:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828439.235748:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790482816:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828439.235796:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828439.235820:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828439/real 1787828439] req@ffff9e680a2d61c0 x1874668790482816/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828439.235841:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790482816/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828439.235857:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828439.235864:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828439.235871:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790482816:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828439.235884:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790482944:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828439.235927:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828439.235951:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828439/real 1787828439] req@ffff9e680a2d4b40 x1874668790482944/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828439.235993:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790482944/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828439.236011:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828439.236018:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828439.236025:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790482944:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828439.236063:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790483072:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828439.236106:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828439.236131:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828439/real 1787828439] req@ffff9e680a2d4f00 x1874668790483072/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828439.236157:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790483072/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828439.236174:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828439.236181:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828439.236187:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790483072:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828439.745903:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828439.937976:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828440.769820:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828440.961877:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828441.793870:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828441.985970:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828442.818315:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828443.009953:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828443.841804:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828444.033945:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828444.353827:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828444.353839:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828444.353844:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828444.353855:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5081 00000100:00080000:3.0:1787828444.353868:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828444.353876:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828444.353913:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790483200/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828444.353939:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828444.353943:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828444.353945:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828444.353950:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5081 00000100:00080000:3.0:1787828444.353954:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828444.353959:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828444.353970:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828444.353987:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790483200:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828444.353988:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790483328/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828444.354007:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828444.354011:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828444.354014:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828444.354019:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5081 00000100:00080000:3.0:1787828444.354024:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828444.354029:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828444.354035:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828444.354058:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790483456/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828444.354070:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828444.354074:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828444.354077:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828444.354081:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828444.354087:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5081 00000100:00100000:2.0:1787828444.354117:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828444/real 1787828444] req@ffff9e680a2d4780 x1874668790483200/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828444.354135:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790483200/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828444.354143:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828444.354147:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828444.354150:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790483200:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787828444.354173:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790483328:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828444.354222:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828444.354230:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787828444.354235:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787828444.354236:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828444.354253:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790483584/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828444.354266:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828444.354270:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828444.354272:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828444/real 1787828444] req@ffff9e680a2d5e00 x1874668790483328/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828444.354273:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828444.354282:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5081 00000100:00080000:3.0:1787828444.354290:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828444.354296:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828444.354301:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790483328/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828444.354305:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828444.354310:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828444.354314:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828444.354317:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790483328:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828444.354325:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790483456:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828444.354325:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790483712/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828444.354345:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828444.354354:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828444/real 1787828444] req@ffff9e680a2d7480 x1874668790483456/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828444.354371:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790483456/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828444.354377:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828444.354380:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828444.354382:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790483456:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828444.354395:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790483584:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828444.354413:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828444.354420:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828444/real 1787828444] req@ffff9e680a2d7c00 x1874668790483584/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828444.354428:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790483584/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828444.354437:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828444.354443:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828444.354451:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790483584:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787828444.354459:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828444.354463:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828444.354465:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828444.354467:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790483712:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787828444.354473:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5081 00000100:00080000:3.0:1787828444.354478:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828444.354495:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828444.354505:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828444/real 1787828444] req@ffff9e680a2d6d00 x1874668790483712/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828444.354517:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828444.354532:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790483712/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828444.354541:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828444.354545:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828444.354549:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828444.354552:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790483712:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828444.354557:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790483840/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828444.354576:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828444.354579:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828444.354581:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828444.354585:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5081 00000100:00080000:3.0:1787828444.354588:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828444.354593:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828444.354597:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828444.354606:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790483968/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828444.354611:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790483840:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828444.354628:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828444.354632:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828444.354635:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828444.354641:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5081 00000100:00080000:3.0:1787828444.354646:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828444.354650:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828444.354655:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828444.354671:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828444.354681:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828444/real 1787828444] req@ffff9e680a2d52c0 x1874668790483840/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828444.354689:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790484096/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828444.354740:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790483840/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828444.354740:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828444.354750:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828444.354755:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828444.354760:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828444.354763:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5081 00000100:00080000:3.0:1787828444.354769:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828444.354771:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828444.354778:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790483840:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828444.354778:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828444.354789:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828444.354797:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790483968:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828444.354805:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790484224/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828444.354820:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828444.354837:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828444/real 1787828444] req@ffff9e680a2d6d00 x1874668790483968/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828444.354851:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790483968/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828444.354860:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828444.354865:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828444.354869:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790483968:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828444.354876:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790484096:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828444.354894:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828444.354902:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828444/real 1787828444] req@ffff9e680a2d7c00 x1874668790484096/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828444.354910:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790484096/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828444.354917:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828444.354920:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828444.354924:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790484096:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828444.354938:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790484224:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828444.354958:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828444.354970:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828444/real 1787828444] req@ffff9e680a2d7480 x1874668790484224/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828444.354982:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790484224/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828444.354991:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828444.354994:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828444.354998:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790484224:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828444.865848:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828445.057988:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828445.889903:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828446.081919:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828446.913800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828447.105841:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828447.937800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828448.129865:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828448.961947:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828449.153924:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828449.473778:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828449.473787:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828449.473790:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828449.473798:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5086 00000100:00080000:3.0:1787828449.473808:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828449.473814:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828449.473886:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790484352/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828449.473921:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828449.473926:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828449.473929:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828449.473935:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5086 00000100:00080000:3.0:1787828449.473939:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828449.473944:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828449.473956:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828449.473971:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790484480/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828449.473979:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790484352:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828449.473997:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828449.474002:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828449.474004:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828449.474028:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5086 00000100:00080000:3.0:1787828449.474031:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828449.474035:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828449.474040:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828449.474053:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790484608/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828449.474063:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828449.474066:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1787828449.474071:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828449.474072:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828449.474079:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5086 00000100:00080000:3.0:1787828449.474085:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828449.474090:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828449.474095:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828449.474106:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790484736/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828449.474113:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828449.474115:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787828449.474118:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828449/real 1787828449] req@ffff9e680a2d5e00 x1874668790484352/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828449.474119:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828449.474127:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5086 00000100:00080000:3.0:1787828449.474132:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828449.474139:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828449.474146:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828449.474156:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790484864/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828449.474159:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790484352/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828449.474174:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828449.474178:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828449.474179:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828449.474180:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828449.474186:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828449.474187:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5086 00000100:00080000:3.0:1787828449.474192:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828449.474193:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790484352:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828449.474199:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828449.474205:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828449.474218:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790484992/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828449.474222:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790484480:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828449.474245:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828449.474249:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828449.474251:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787828449.474256:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828449.474259:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5086 00000100:00080000:3.0:1787828449.474264:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828449.474268:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828449.474273:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828449.474302:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790485120/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828449.474306:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828449/real 1787828449] req@ffff9e680a2d4780 x1874668790484480/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828449.474367:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828449.474378:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790484480/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828449.474380:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828449.474388:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828449.474400:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828449.474406:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5086 00000100:00080000:0.0:1787828449.474411:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828449.474420:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828449.474421:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790484480:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828449.474435:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828449.474441:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790484608:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787828449.474446:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828449.474480:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828449.474505:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790485248/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828449.474532:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828449.474535:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828449/real 1787828449] req@ffff9e680a2d4f00 x1874668790484608/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828449.474537:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828449.474540:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828449.474548:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5086 00000100:00080000:3.0:1787828449.474552:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828449.474555:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790484608/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828449.474559:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828449.474567:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787828449.474577:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828449.474581:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790485376/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828449.474582:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828449.474590:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790484608:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828449.474601:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790484736:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828449.474620:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828449.474627:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828449/real 1787828449] req@ffff9e680a2d4b40 x1874668790484736/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828449.474640:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790484736/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828449.474648:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828449.474651:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828449.474654:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790484736:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828449.474661:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790484864:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828449.474692:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828449.474703:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828449/real 1787828449] req@ffff9e680a2d61c0 x1874668790484864/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828449.474733:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790484864/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828449.474739:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828449.474741:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828449.474744:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790484864:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828449.474758:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790484992:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828449.474782:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828449.474789:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828449/real 1787828449] req@ffff9e680a2d70c0 x1874668790484992/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828449.474796:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790484992/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828449.474803:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828449.474805:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828449.474807:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790484992:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828449.474810:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790485120:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828449.474838:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828449.474849:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828449/real 1787828449] req@ffff9e680a2d5a40 x1874668790485120/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828449.474861:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790485120/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828449.474869:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828449.474872:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828449.474875:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790485120:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828449.474882:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790485248:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828449.474899:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828449.474908:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828449/real 1787828449] req@ffff9e680a2d43c0 x1874668790485248/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828449.474920:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790485248/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828449.474929:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828449.474932:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828449.474935:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790485248:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828449.474940:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790485376:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828449.474955:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828449.474964:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828449/real 1787828449] req@ffff9e680a2d6580 x1874668790485376/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828449.474975:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790485376/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828449.474983:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828449.474986:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828449.474989:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790485376:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828449.985841:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828450.177918:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828451.010412:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828451.202014:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828452.034053:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828452.225946:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828453.057875:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828453.250084:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828454.082387:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828454.273932:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828454.593804:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828454.593818:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828454.593823:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828454.593835:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5091 00000100:00080000:1.0:1787828454.593848:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828454.593858:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828454.593899:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790485504/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828454.593947:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828454.593952:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828454.593955:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828454.593961:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5091 00000100:00080000:1.0:1787828454.593966:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828454.593971:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828454.593985:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828454.594017:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790485632/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828454.594027:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790485504:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828454.594033:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828454.594038:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828454.594039:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828454.594043:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5091 00000100:00080000:1.0:1787828454.594046:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828454.594051:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828454.594055:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828454.594094:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790485760/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828454.594107:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787828454.594112:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828454.594117:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828454.594122:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828454.594129:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5091 00000100:00080000:1.0:1787828454.594134:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828454.594141:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828454.594148:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828454.594164:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828454/real 1787828454] req@ffff9e6831aaad00 x1874668790485504/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828454.594172:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790485888/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828454.594188:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828454.594191:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828454.594195:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828454.594197:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790485504/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828454.594200:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5091 00000100:00080000:1.0:1787828454.594206:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828454.594218:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787828454.594221:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787828454.594222:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787828454.594232:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828454.594237:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790486016/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828454.594242:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790485504:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828454.594259:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828454.594264:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828454.594266:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828454.594273:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5091 00000100:00100000:0.0:1787828454.594274:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790485632:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828454.594282:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828454.594292:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828454.594301:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828454.594306:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828454.594320:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790486144/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828454.594323:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828454/real 1787828454] req@ffff9e6831aa8000 x1874668790485632/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828454.594369:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828454.594375:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790485632/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828454.594376:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828454.594381:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828454.594409:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5091 00000100:00080000:0.0:1787828454.594411:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828454.594414:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828454.594418:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828454.594422:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828454.594424:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790485632:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787828454.594430:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828454.594434:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790485760:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828454.594445:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790486272/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828454.594471:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787828454.594477:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828454.594478:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828454.594483:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828454.594488:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5091 00000100:00080000:1.0:1787828454.594493:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828454.594497:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828454.594503:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828454/real 1787828454] req@ffff9e6831aa9a40 x1874668790485760/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828454.594505:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828454.594526:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790486400/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828454.594530:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790485760/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828454.594558:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828454.594559:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828454.594565:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828454.594570:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828454.594571:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828454.594580:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790485760:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828454.594582:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5091 00000100:00080000:1.0:1787828454.594593:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828454.594600:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790485888:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787828454.594604:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828454.594615:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828454.594635:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790486528/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828454.594636:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828454.594649:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828454/real 1787828454] req@ffff9e6831aab480 x1874668790485888/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828454.594661:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790485888/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828454.594669:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828454.594672:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828454.594674:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790485888:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828454.594680:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790486016:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828454.594718:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828454.594728:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828454/real 1787828454] req@ffff9e6831aabc00 x1874668790486016/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828454.594737:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790486016/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828454.594761:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828454.594763:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828454.594766:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790486016:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828454.594783:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790486144:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828454.594801:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828454.594810:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828454/real 1787828454] req@ffff9e6831aaad00 x1874668790486144/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828454.594820:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790486144/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828454.594829:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828454.594831:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828454.594834:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790486144:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828454.594838:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790486272:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828454.594860:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828454.594869:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828454/real 1787828454] req@ffff9e6831aaa1c0 x1874668790486272/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828454.594878:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790486272/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828454.594885:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828454.594888:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828454.594891:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790486272:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828454.594896:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790486400:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828454.594910:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828454.594916:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828454/real 1787828454] req@ffff9e6831aa8780 x1874668790486400/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828454.594926:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790486400/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828454.594938:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828454.594941:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828454.594944:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790486400:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828454.594948:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790486528:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828454.594960:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828454.594967:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828454/real 1787828454] req@ffff9e6831aaa940 x1874668790486528/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828454.594976:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790486528/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828454.594982:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828454.594984:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828454.594987:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790486528:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828455.105850:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828455.298007:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828456.129902:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828456.321837:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828457.154022:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828457.346193:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828458.177823:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828458.369960:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828459.201898:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828459.393843:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828459.713919:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828459.713937:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828459.713944:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828459.713961:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5096 00000100:00080000:1.0:1787828459.713979:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828459.713994:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828459.714060:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790486656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828459.714135:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828459.714145:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828459.714149:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828459.714158:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5096 00000100:00080000:1.0:1787828459.714167:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828459.714176:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828459.714193:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828459.714223:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790486784/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828459.714246:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828459.714252:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828459.714258:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828459.714265:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5096 00000100:00080000:1.0:1787828459.714281:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828459.714290:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828459.714299:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828459.714320:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790486912/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828459.714338:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828459.714344:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828459.714349:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828459.714356:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5096 00000100:00080000:1.0:1787828459.714363:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828459.714370:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828459.714381:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828459.714401:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790487040/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828459.714447:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828459.714453:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828459.714457:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828459.714464:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5096 00000100:00080000:1.0:1787828459.714471:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828459.714478:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828459.714486:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828459.714507:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790487168/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828459.714525:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828459.714531:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828459.714534:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828459.714559:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5096 00000100:00080000:1.0:1787828459.714566:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828459.714573:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828459.714581:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828459.714600:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790487296/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828459.714632:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828459.714637:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828459.714641:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828459.714647:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5096 00000100:00080000:1.0:1787828459.714653:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828459.714658:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828459.714666:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828459.714683:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790487424/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828459.714697:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828459.714702:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828459.714704:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828459.714710:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5096 00000100:00080000:1.0:1787828459.714716:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828459.714722:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828459.714762:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828459.714799:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790487552/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828459.714818:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828459.714824:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828459.714827:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828459.714834:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5096 00000100:00080000:1.0:1787828459.714841:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828459.714847:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828459.714855:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828459.714875:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790487680/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828459.715279:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790486656:192.168.203.109@tcp:250:kworker.0 00000800:00000100:0.0:1787828459.715376:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828459.715447:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828459/real 1787828459] req@ffff9e6831aa9e00 x1874668790486656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828459.715472:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790486656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828459.715485:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828459.715490:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828459.715495:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790486656:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787828459.715504:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790486784:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828459.715535:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828459.715552:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828459/real 1787828459] req@ffff9e6831aa8f00 x1874668790486784/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828459.715573:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790486784/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828459.715605:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828459.715610:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828459.715615:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790486784:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828459.715626:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790486912:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828459.715659:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828459.715696:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828459/real 1787828459] req@ffff9e6831aaa580 x1874668790486912/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828459.715720:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790486912/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828459.715788:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828459.715793:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828459.715798:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790486912:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828459.715806:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790487040:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828459.715837:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828459.715852:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828459/real 1787828459] req@ffff9e6831aa9680 x1874668790487040/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828459.715867:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790487040/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828459.715886:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828459.715889:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828459.715893:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790487040:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828459.715901:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790487168:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828459.715923:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828459.715934:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828459/real 1787828459] req@ffff9e6831aa8b40 x1874668790487168/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828459.715947:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790487168/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828459.715956:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828459.715960:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828459.715963:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790487168:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828459.715969:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790487296:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828459.716004:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828459.716014:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828459/real 1787828459] req@ffff9e6831aaa940 x1874668790487296/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828459.716025:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790487296/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828459.716042:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828459.716045:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828459.716048:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790487296:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828459.716053:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790487424:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828459.716069:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828459.716077:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828459/real 1787828459] req@ffff9e6831aa8780 x1874668790487424/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828459.716088:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790487424/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828459.716096:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828459.716099:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828459.716101:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790487424:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828459.716107:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790487552:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828459.716122:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828459.716131:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828459/real 1787828459] req@ffff9e6831aaa1c0 x1874668790487552/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828459.716141:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790487552/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828459.716149:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828459.716151:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828459.716155:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790487552:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828459.716161:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790487680:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828459.716176:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828459.716184:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828459/real 1787828459] req@ffff9e6831aaad00 x1874668790487680/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828459.716195:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790487680/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828459.716203:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828459.716205:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828459.716208:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790487680:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828460.225912:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828460.417816:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828461.249936:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828461.441835:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828462.273816:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828462.465831:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828463.297846:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828463.489919:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828464.321957:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828464.513898:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828464.834867:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828464.834883:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828464.834890:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828464.834921:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5101 00000100:00080000:3.0:1787828464.834936:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828464.834944:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828464.834992:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790487808/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828464.835029:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828464.835034:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828464.835036:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828464.835041:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5101 00000100:00080000:3.0:1787828464.835047:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828464.835053:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828464.835066:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828464.835081:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790487808:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828464.835088:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790487936/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828464.835118:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828464.835121:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828464.835123:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828464.835127:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5101 00000100:00080000:3.0:1787828464.835131:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828464.835134:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828464.835139:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828464.835150:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790488064/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828464.835159:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828464.835162:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828464.835164:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828464.835170:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5101 00000100:00080000:3.0:1787828464.835186:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828464.835192:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828464.835199:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828464.835214:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790488192/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828464.835220:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828464.835234:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828464.835240:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828464.835243:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828464.835249:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5101 00000100:00080000:3.0:1787828464.835255:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828464.835259:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828464.835264:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828464.835278:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790488320/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828464.835286:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828464.835289:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828464/real 1787828464] req@ffff9e680a2d4000 x1874668790487808/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828464.835289:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828464.835293:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828464.835301:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5101 00000100:00080000:3.0:1787828464.835307:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828464.835314:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828464.835319:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828464.835332:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790487808/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828464.835333:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790488448/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828464.835399:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828464.835400:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828464.835404:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828464.835408:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828464.835408:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828464.835418:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5101 00000100:00100000:2.0:1787828464.835419:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790487808:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828464.835440:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828464.835446:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828464.835451:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828464.835461:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790488576/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828464.835469:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790487936:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828464.835474:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828464.835478:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828464.835480:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828464.835485:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5101 00000100:00080000:3.0:1787828464.835489:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828464.835495:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787828464.835502:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787828464.835503:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828464.835517:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828464/real 1787828464] req@ffff9e680a2d6580 x1874668790487936/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828464.835525:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790488704/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828464.835536:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790487936/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828464.835548:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828464.835550:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828464.835558:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828464.835559:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828464.835563:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790487936:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828464.835566:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828464.835572:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790488064:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828464.835576:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5101 00000100:00080000:3.0:1787828464.835583:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828464.835587:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828464.835593:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828464.835599:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828464.835610:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790488832/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828464.835620:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828464/real 1787828464] req@ffff9e680a2d43c0 x1874668790488064/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828464.835660:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790488064/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828464.835668:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828464.835671:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828464.835673:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790488064:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828464.835678:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790488192:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828464.835697:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828464.835705:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828464/real 1787828464] req@ffff9e680a2d5a40 x1874668790488192/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828464.835716:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790488192/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828464.835741:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828464.835744:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828464.835747:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790488192:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828464.835752:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790488320:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828464.835765:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828464.835786:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828464/real 1787828464] req@ffff9e680a2d70c0 x1874668790488320/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828464.835797:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790488320/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828464.835809:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828464.835811:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828464.835813:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790488320:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828464.835817:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790488448:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828464.835829:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828464.835836:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828464/real 1787828464] req@ffff9e680a2d61c0 x1874668790488448/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828464.835843:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790488448/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828464.835851:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828464.835853:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828464.835855:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790488448:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828464.835871:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790488576:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828464.835882:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828464.835888:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828464/real 1787828464] req@ffff9e680a2d4b40 x1874668790488576/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828464.835896:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790488576/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828464.835901:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828464.835903:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828464.835906:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790488576:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828464.835918:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790488704:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828464.835943:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828464.835956:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828464/real 1787828464] req@ffff9e680a2d4f00 x1874668790488704/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828464.835968:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790488704/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828464.835975:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828464.835977:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828464.835980:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790488704:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828464.835984:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790488832:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828464.835999:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828464.836007:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828464/real 1787828464] req@ffff9e680a2d4780 x1874668790488832/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828464.836015:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790488832/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828464.836020:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828464.836021:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828464.836023:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790488832:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828465.345854:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828465.538005:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828466.369943:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828466.561924:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828467.393817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828467.585954:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828468.417837:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828468.609922:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828469.441765:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828469.633902:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828469.953841:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828469.953858:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828469.953864:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828469.953875:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5106 00000100:00080000:1.0:1787828469.953887:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828469.953894:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828469.953939:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790488960/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828469.953966:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828469.953969:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828469.953970:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828469.953975:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5106 00000100:00080000:1.0:1787828469.953979:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828469.953983:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828469.953995:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828469.954011:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790489088/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828469.954034:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790488960:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828469.954035:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828469.954043:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828469.954045:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828469.954049:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5106 00000100:00080000:1.0:1787828469.954052:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828469.954056:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828469.954061:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828469.954070:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790489216/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828469.954078:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828469.954082:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828469.954084:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828469.954088:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5106 00000100:00080000:1.0:1787828469.954093:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828469.954097:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828469.954102:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828469.954116:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790489344/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828469.954131:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828469.954148:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828469.954152:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828469.954154:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828469.954158:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5106 00000100:00080000:1.0:1787828469.954161:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828469.954165:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828469.954170:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828469.954182:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790489472/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828469.954187:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828469/real 1787828469] req@ffff9e6831aabc00 x1874668790488960/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828469.954204:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828469.954211:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828469.954213:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828469.954220:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5106 00000100:00080000:1.0:1787828469.954222:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828469.954225:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828469.954229:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790488960/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828469.954230:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828469.954241:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790489600/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828469.954243:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828469.954252:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828469.954254:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828469.954258:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790488960:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828469.954259:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828469.954263:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828469.954268:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5106 00000100:00080000:1.0:1787828469.954270:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828469.954273:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828469.954277:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828469.954282:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790489728/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828469.954288:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790489088:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828469.954292:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828469.954296:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828469.954298:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828469.954302:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5106 00000100:00080000:1.0:1787828469.954316:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828469.954320:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787828469.954324:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1787828469.954327:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828469.954337:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828469/real 1787828469] req@ffff9e6831aab480 x1874668790489088/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828469.954339:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790489856/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828469.954377:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828469.954381:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828469.954383:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828469.954384:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790489088/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828469.954391:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5106 00000100:00080000:2.0:1787828469.954398:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828469.954399:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828469.954405:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828469.954409:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828469.954410:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790489088:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828469.954419:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790489216:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787828469.954420:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828469.954439:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790489984/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828469.954446:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828469.954470:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828469/real 1787828469] req@ffff9e6831aa9a40 x1874668790489216/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828469.954481:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790489216/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828469.954490:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828469.954494:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828469.954499:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790489216:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828469.954507:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790489344:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828469.954541:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828469.954551:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828469/real 1787828469] req@ffff9e6831aa8000 x1874668790489344/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828469.954561:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790489344/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828469.954570:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828469.954573:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828469.954577:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790489344:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828469.954584:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790489472:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828469.954606:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828469.954617:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828469/real 1787828469] req@ffff9e6831aaad00 x1874668790489472/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828469.954632:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790489472/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828469.954637:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828469.954640:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828469.954642:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790489472:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828469.954645:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790489600:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828469.954664:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828469.954670:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828469/real 1787828469] req@ffff9e6831aaa1c0 x1874668790489600/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828469.954677:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790489600/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828469.954684:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828469.954685:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828469.954687:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790489600:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828469.954725:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790489728:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828469.954739:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828469.954746:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828469/real 1787828469] req@ffff9e6831aa8780 x1874668790489728/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828469.954754:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790489728/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828469.954764:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828469.954766:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828469.954768:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790489728:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828469.954771:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790489856:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828469.954781:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828469.954787:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828469/real 1787828469] req@ffff9e6831aaa940 x1874668790489856/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828469.954794:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790489856/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828469.954800:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828469.954801:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828469.954803:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790489856:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828469.954806:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790489984:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828469.954816:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828469.954821:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828469/real 1787828469] req@ffff9e6831aa8b40 x1874668790489984/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828469.954828:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790489984/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828494 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828469.954833:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828469.954835:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828469.954837:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790489984:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828470.465816:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828470.658014:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828471.489895:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828471.682019:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828472.513859:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828472.705995:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828473.537903:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828473.730063:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828474.561828:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828474.753854:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828475.073828:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828475.073851:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828475.073857:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828475.073871:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5111 00000100:00080000:1.0:1787828475.073894:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828475.073905:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828475.073955:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790490112/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828475.074008:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828475.074011:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828475.074013:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828475.074016:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5111 00000100:00080000:1.0:1787828475.074020:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828475.074023:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828475.074034:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828475.074068:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790490240/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828475.074077:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790490112:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828475.074088:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828475.074093:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828475.074095:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828475.074100:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5111 00000100:00080000:1.0:1787828475.074121:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828475.074125:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828475.074131:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828475.074141:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790490368/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828475.074149:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828475.074152:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828475.074154:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828475.074158:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5111 00000100:00080000:1.0:1787828475.074161:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828475.074165:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828475.074187:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828475.074198:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828475.074200:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790490496/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828475.074223:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828475.074228:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828475.074230:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828475.074236:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5111 00000100:00080000:1.0:1787828475.074247:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828475.074252:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828475.074261:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828475.074284:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790490624/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828475.074288:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828475/real 1787828475] req@ffff9e6831aa9680 x1874668790490112/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828475.074370:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828475.074375:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828475.074379:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828475.074384:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5111 00000100:00080000:1.0:1787828475.074388:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828475.074390:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790490112/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828475.074398:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828475.074414:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828475.074418:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828475.074428:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828475.074430:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790490752/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828475.074442:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790490112:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828475.074467:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828475.074471:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828475.074473:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828475.074478:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5111 00000100:00080000:1.0:1787828475.074484:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828475.074490:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828475.074490:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790490240:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787828475.074498:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828475.074514:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790490880/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828475.074523:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828475.074533:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828475.074536:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828475.074539:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828475.074546:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5111 00000100:00080000:1.0:1787828475.074556:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828475.074559:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828475/real 1787828475] req@ffff9e6831aaa580 x1874668790490240/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828475.074564:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828475.074575:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828475.074608:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790491008/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828475.074609:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790490240/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828475.074688:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828475.074691:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828475.074698:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828475.074701:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828475.074703:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828475.074713:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5111 00000100:00080000:1.0:1787828475.074719:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828475.074748:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790490240:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828475.074757:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790490368:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828475.074768:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828475.074775:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828475.074783:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828475.074815:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790491136/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828475.074816:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828475/real 1787828475] req@ffff9e6831aa8f00 x1874668790490368/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828475.074874:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790490368/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828475.074885:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828475.074889:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828475.074892:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790490368:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828475.074898:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790490496:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828475.074918:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828475.074927:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828475/real 1787828475] req@ffff9e6831aa9e00 x1874668790490496/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828475.074938:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790490496/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828475.074945:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828475.074949:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828475.074951:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790490496:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828475.074968:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790490624:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828475.074981:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828475.074990:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828475/real 1787828475] req@ffff9e6831aa8b40 x1874668790490624/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828475.075001:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790490624/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828475.075009:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828475.075011:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828475.075014:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790490624:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828475.075019:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790490752:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828475.075033:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828475.075047:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828475/real 1787828475] req@ffff9e6831aaa940 x1874668790490752/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828475.075058:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790490752/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828475.075067:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828475.075069:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828475.075072:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790490752:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828475.075091:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790490880:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828475.075105:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828475.075114:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828475/real 1787828475] req@ffff9e6831aa8780 x1874668790490880/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828475.075124:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790490880/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828475.075146:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828475.075149:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828475.075152:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790490880:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828475.075157:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790491008:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828475.075171:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828475.075179:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828475/real 1787828475] req@ffff9e6831aaa1c0 x1874668790491008/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828475.075189:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790491008/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828475.075222:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828475.075225:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828475.075228:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790491008:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828475.075234:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790491136:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828475.075252:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828475.075262:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828475/real 1787828475] req@ffff9e6831aaad00 x1874668790491136/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828475.075273:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790491136/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828475.075280:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828475.075283:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828475.075287:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790491136:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828475.585855:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828475.777892:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828476.609845:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828476.801908:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828477.633844:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828477.825984:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828478.657800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828478.849867:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828479.681835:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828479.874012:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828480.193806:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828480.193817:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828480.193822:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828480.193832:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5116 00000100:00080000:1.0:1787828480.193843:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828480.193850:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828480.193883:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790491264/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828480.193921:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828480.193926:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828480.193929:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828480.193935:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5116 00000100:00080000:1.0:1787828480.193940:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828480.193945:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828480.193957:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828480.193975:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790491264:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828480.193980:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790491392/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828480.194000:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828480.194005:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828480.194007:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828480.194011:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5116 00000100:00080000:1.0:1787828480.194016:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828480.194020:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828480.194026:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828480.194037:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790491520/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828480.194053:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828480.194056:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828480.194057:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828480.194060:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5116 00000100:00080000:1.0:1787828480.194063:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787828480.194068:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828480.194070:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828480.194076:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828480.194087:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790491648/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828480.194096:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828480.194099:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828480.194101:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828480.194105:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5116 00000100:00080000:1.0:1787828480.194170:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828480.194175:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828480.194179:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828480.194191:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790491776/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828480.194209:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828480.194211:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828480/real 1787828480] req@ffff9e6831aa8000 x1874668790491264/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828480.194212:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828480.194215:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828480.194221:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5116 00000100:00080000:1.0:1787828480.194225:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828480.194232:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828480.194237:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828480.194247:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790491904/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828480.194255:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790491264/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828480.194256:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828480.194265:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828480.194267:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828480.194294:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5116 00000100:00080000:1.0:1787828480.194300:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828480.194302:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828480.194304:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828480.194313:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787828480.194316:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828480.194326:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790492032/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828480.194336:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790491264:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828480.194354:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828480.194356:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828480.194358:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828480.194361:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5116 00000100:00080000:1.0:1787828480.194363:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828480.194365:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828480.194368:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828480.194374:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790492160/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828480.194380:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828480.194382:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828480.194384:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828480.194386:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5116 00000100:00080000:1.0:1787828480.194388:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828480.194391:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828480.194394:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828480.194400:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790492288/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828480.194510:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790491392:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828480.194628:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828480.194668:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828480/real 1787828480] req@ffff9e6831aa9a40 x1874668790491392/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828480.194687:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790491392/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828480.194726:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828480.194734:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828480.194798:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790491392:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828480.194817:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790491520:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828480.194863:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828480.194889:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828480/real 1787828480] req@ffff9e6831aab480 x1874668790491520/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828480.194916:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790491520/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828480.194935:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828480.194942:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828480.194948:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790491520:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828480.194962:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790491648:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828480.195013:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828480.195028:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828480/real 1787828480] req@ffff9e6831aabc00 x1874668790491648/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828480.195042:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790491648/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828480.195052:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828480.195056:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828480.195060:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790491648:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828480.195067:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790491776:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828480.195116:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828480.195131:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828480/real 1787828480] req@ffff9e6831aaad00 x1874668790491776/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828480.195165:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790491776/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828480.195182:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828480.195189:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828480.195195:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790491776:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828480.195207:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790491904:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828480.195252:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828480.195315:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828480/real 1787828480] req@ffff9e6831aaa1c0 x1874668790491904/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828480.195336:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790491904/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828480.195367:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828480.195370:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828480.195374:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790491904:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828480.195381:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790492032:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828480.195404:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828480.195418:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828480/real 1787828480] req@ffff9e6831aa8780 x1874668790492032/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828480.195431:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790492032/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828480.195441:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828480.195444:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828480.195448:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790492032:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828480.195455:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790492160:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828480.195475:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828480.195486:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828480/real 1787828480] req@ffff9e6831aaa940 x1874668790492160/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828480.195510:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790492160/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828480.195520:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828480.195523:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828480.195528:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790492160:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828480.195534:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790492288:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828480.195557:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828480.195570:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828480/real 1787828480] req@ffff9e6831aa8b40 x1874668790492288/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828480.195584:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790492288/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828480.195600:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828480.195604:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828480.195608:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790492288:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828480.705812:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828480.897886:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828481.729883:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828481.921854:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828482.753856:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828482.945885:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828483.777926:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828483.969873:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828484.801812:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828484.993978:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828485.313884:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828485.313894:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828485.313898:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828485.313907:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5122 00000100:00080000:3.0:1787828485.313917:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828485.313923:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828485.313960:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790492416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828485.313992:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828485.313995:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828485.313997:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828485.314001:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5122 00000100:00080000:3.0:1787828485.314003:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828485.314006:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828485.314015:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828485.314026:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790492544/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828485.314037:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790492416:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828485.314038:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828485.314047:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828485.314051:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828485.314055:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5122 00000100:00080000:3.0:1787828485.314057:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828485.314060:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828485.314064:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828485.314073:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790492672/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828485.314081:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828485.314084:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828485.314086:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828485.314089:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5122 00000100:00080000:3.0:1787828485.314095:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828485.314099:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828485.314104:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828485.314110:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828485.314115:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790492800/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828485.314126:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828485.314129:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828485.314130:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828485.314133:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5122 00000100:00080000:3.0:1787828485.314136:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828485.314139:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828485.314143:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828485.314155:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828485/real 1787828485] req@ffff9e680a2d5e00 x1874668790492416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828485.314157:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790492928/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828485.314196:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828485.314200:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828485.314202:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828485.314207:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790492416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828485.314211:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5122 00000100:00080000:3.0:1787828485.314218:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828485.314226:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787828485.314228:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828485.314231:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828485.314252:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828485.314259:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790492416:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828485.314264:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790493056/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828485.314279:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828485.314282:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828485.314301:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828485.314308:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790492544:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828485.314308:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5122 00000100:00080000:3.0:1787828485.314318:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828485.314324:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828485.314329:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828485.314341:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790493184/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828485.314348:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828485.314362:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828485.314366:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787828485.314369:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828485/real 1787828485] req@ffff9e680a2d7480 x1874668790492544/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828485.314369:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828485.314380:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5122 00000100:00080000:3.0:1787828485.314388:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828485.314394:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828485.314398:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790492544/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828485.314401:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828485.314417:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828485.314419:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790493312/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828485.314430:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828485.314435:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828485.314442:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790492544:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828485.314442:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828485.314444:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828485.314473:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5122 00000100:00080000:3.0:1787828485.314477:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828485.314480:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790492672:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828485.314485:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828485.314495:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828485.314506:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790493440/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828485.314522:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828485.314559:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828485/real 1787828485] req@ffff9e680a2d7c00 x1874668790492672/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828485.314576:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790492672/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828485.314587:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828485.314591:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828485.314595:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790492672:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828485.314618:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790492800:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828485.314638:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828485.314647:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828485/real 1787828485] req@ffff9e680a2d6d00 x1874668790492800/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828485.314659:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790492800/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828485.314667:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828485.314670:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828485.314673:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790492800:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828485.314678:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790492928:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828485.314693:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828485.314706:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828485/real 1787828485] req@ffff9e680a2d52c0 x1874668790492928/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828485.314738:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790492928/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828485.314750:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828485.314754:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828485.314758:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790492928:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828485.314766:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790493056:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828485.314815:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828485.314830:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828485/real 1787828485] req@ffff9e680a2d4780 x1874668790493056/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828485.314846:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790493056/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828485.314859:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828485.314863:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828485.314868:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790493056:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828485.314892:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790493184:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828485.314912:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828485.314921:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828485/real 1787828485] req@ffff9e680a2d4f00 x1874668790493184/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828485.314931:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790493184/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828485.314937:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828485.314940:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828485.314943:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790493184:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828485.314948:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790493312:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828485.314960:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828485.314967:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828485/real 1787828485] req@ffff9e680a2d4b40 x1874668790493312/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828485.314975:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790493312/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828485.314982:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828485.314984:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828485.314986:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790493312:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828485.314990:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790493440:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828485.315002:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828485.315009:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828485/real 1787828485] req@ffff9e680a2d61c0 x1874668790493440/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828485.315018:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790493440/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828485.315024:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828485.315026:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828485.315028:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790493440:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828485.825867:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828486.017965:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828486.849839:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828487.041952:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828487.873880:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828488.065939:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828488.897865:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828489.089950:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828489.921813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828490.113960:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828490.433870:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828490.433884:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828490.433899:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828490.433912:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5127 00000100:00080000:3.0:1787828490.433925:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828490.433935:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828490.434016:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790493568/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828490.434069:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828490.434077:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828490.434080:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828490.434089:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5127 00000100:00080000:3.0:1787828490.434097:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828490.434105:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828490.434121:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828490.434149:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790493696/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828490.434187:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828490.434194:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790493568:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828490.434197:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828490.434233:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828490.434243:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5127 00000100:00080000:3.0:1787828490.434253:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828490.434268:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828490.434281:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828490.434302:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790493824/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828490.434317:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828490.434319:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828490.434327:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828490.434333:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828490.434341:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5127 00000100:00080000:3.0:1787828490.434349:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828490.434355:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828490.434362:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828490.434392:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790493952/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828490.434400:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828490/real 1787828490] req@ffff9e680a2d70c0 x1874668790493568/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828490.434441:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828490.434448:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790493568/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828490.434455:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828490.434467:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828490.434472:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828490.434483:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5127 00000100:00080000:2.0:1787828490.434484:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828490.434495:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790493568:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828490.434514:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828490.434524:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828490.434532:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790493696:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787828490.434534:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828490.434571:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790494080/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828490.434588:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828490.434590:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828490.434597:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828490.434601:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828490.434611:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5127 00000100:00100000:2.0:1787828490.434614:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828490/real 1787828490] req@ffff9e680a2d5a40 x1874668790493696/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828490.434618:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828490.434640:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828490.434648:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790493696/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828490.434651:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828490.434676:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790494208/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828490.434678:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828490.434736:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828490.434743:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828490.434758:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828490.434767:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5127 00000100:00080000:2.0:1787828490.434772:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828490.434780:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828490.434786:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790493696:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828490.434798:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828490.434804:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790493824:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787828490.434824:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828490.434858:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790494336/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828490.434869:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828490.434897:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828490/real 1787828490] req@ffff9e680a2d43c0 x1874668790493824/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828490.434910:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828490.434923:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828490.434926:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828490.434933:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790493824/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828490.434940:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5127 00000100:00080000:3.0:1787828490.434954:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828490.434983:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787828490.434988:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828490.434997:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:3.0:1787828490.435000:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828490.435003:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790493824:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828490.435019:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790493952:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828490.435044:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790494464/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828490.435061:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828490.435065:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828490.435076:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828490.435078:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828490/real 1787828490] req@ffff9e680a2d6580 x1874668790493952/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828490.435083:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828490.435104:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5127 00000100:00100000:2.0:1787828490.435112:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790493952/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828490.435117:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828490.435135:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787828490.435143:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828490.435152:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:3.0:1787828490.435154:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828490.435158:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790493952:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828490.435187:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790494592/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828490.435190:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790494080:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828490.435251:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828490.435271:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828490/real 1787828490] req@ffff9e680a2d4000 x1874668790494080/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828490.435287:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790494080/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828490.435298:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828490.435302:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828490.435307:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790494080:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828490.435314:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790494208:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828490.435333:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828490.435345:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828490/real 1787828490] req@ffff9e680a2d70c0 x1874668790494208/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828490.435358:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790494208/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828490.435370:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828490.435373:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828490.435377:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790494208:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828490.435383:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790494336:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828490.435401:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828490.435412:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828490/real 1787828490] req@ffff9e680a2d61c0 x1874668790494336/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828490.435425:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790494336/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828490.435434:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828490.435438:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828490.435441:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790494336:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828490.435452:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790494464:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828490.435470:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828490.435481:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828490/real 1787828490] req@ffff9e680a2d4b40 x1874668790494464/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828490.435495:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790494464/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828490.435505:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828490.435508:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828490.435511:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790494464:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828490.435534:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790494592:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828490.435553:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828490.435564:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828490/real 1787828490] req@ffff9e680a2d4f00 x1874668790494592/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828490.435577:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790494592/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828490.435586:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828490.435590:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828490.435593:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790494592:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828490.945814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828491.137842:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828491.969814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828492.161857:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828492.993814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828493.186381:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828494.017882:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828494.209900:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828495.041813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828495.233898:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828495.553896:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828495.553905:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828495.553909:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828495.553919:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5132 00000100:00080000:1.0:1787828495.553929:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828495.553936:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828495.554000:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790494720/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828495.554040:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828495.554045:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828495.554047:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828495.554052:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5132 00000100:00080000:1.0:1787828495.554056:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828495.554061:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828495.554074:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828495.554088:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790494720:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828495.554096:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790494848/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828495.554112:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828495.554116:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828495.554118:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828495.554122:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5132 00000100:00080000:1.0:1787828495.554126:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828495.554130:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828495.554135:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828495.554148:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790494976/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828495.554158:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828495.554162:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828495.554186:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828495.554194:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5132 00000800:00000100:2.0:1787828495.554198:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828495.554201:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828495.554209:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828495.554216:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828495.554252:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790495104/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828495.554266:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828495.554270:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828495.554273:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828495.554279:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5132 00000100:00100000:2.0:1787828495.554288:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828495/real 1787828495] req@ffff9e6831aa9e00 x1874668790494720/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828495.554290:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828495.554306:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828495.554321:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828495.554336:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790494720/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828495.554343:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790495232/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828495.554409:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828495.554410:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828495.554418:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828495.554421:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828495.554422:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828495.554433:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5132 00000100:00100000:2.0:1787828495.554434:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790494720:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828495.554437:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828495.554444:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828495.554450:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828495.554479:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790495360/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828495.554494:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828495.554503:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790494848:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828495.554504:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828495.554508:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828495.554522:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5132 00000100:00080000:1.0:1787828495.554528:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828495.554533:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828495.554539:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828495.554574:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790495488/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828495.554594:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828495.554600:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828495.554606:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828495.554609:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828495.554615:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5132 00000100:00080000:1.0:1787828495.554621:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828495.554625:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828495/real 1787828495] req@ffff9e6831aa8f00 x1874668790494848/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828495.554632:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828495.554644:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828495.554658:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790494848/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828495.554665:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790495616/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828495.554693:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828495.554696:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828495.554703:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828495.554707:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828495.554713:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790494848:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828495.554715:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828495.554730:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5132 00000100:00100000:2.0:1787828495.554735:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790494976:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828495.554738:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828495.554751:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828495.554758:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828495.554822:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790495744/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828495.554850:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828495.554864:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828495/real 1787828495] req@ffff9e6831aaa580 x1874668790494976/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828495.554875:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790494976/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828495.554883:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828495.554887:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828495.554890:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790494976:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828495.554896:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790495104:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828495.554911:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828495.554919:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828495/real 1787828495] req@ffff9e6831aa9680 x1874668790495104/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828495.554929:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790495104/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828495.554936:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828495.554938:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828495.554941:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790495104:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828495.554945:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790495232:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828495.554977:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828495.554988:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828495/real 1787828495] req@ffff9e6831aa8b40 x1874668790495232/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828495.555001:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790495232/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828495.555010:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828495.555013:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828495.555017:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790495232:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828495.555053:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790495360:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828495.555076:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828495.555088:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828495/real 1787828495] req@ffff9e6831aaa940 x1874668790495360/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828495.555103:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790495360/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828495.555115:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828495.555118:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828495.555122:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790495360:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828495.555129:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790495488:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828495.555147:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828495.555157:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828495/real 1787828495] req@ffff9e6831aa8780 x1874668790495488/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828495.555225:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790495488/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828495.555235:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828495.555239:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828495.555243:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790495488:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828495.555250:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790495616:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828495.555295:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828495.555310:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828495/real 1787828495] req@ffff9e6831aaa1c0 x1874668790495616/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828495.555324:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790495616/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828495.555342:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828495.555346:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828495.555350:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790495616:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828495.555357:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790495744:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828495.555376:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828495.555387:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828495/real 1787828495] req@ffff9e6831aaad00 x1874668790495744/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828495.555418:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790495744/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828495.555425:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828495.555427:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828495.555429:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790495744:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828496.065797:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828496.257940:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828497.089818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828497.282261:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828498.113793:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828498.305952:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828499.137793:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828499.329868:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828500.161757:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828500.353858:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828500.673898:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828500.673906:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828500.673911:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828500.673918:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5137 00000100:00080000:3.0:1787828500.673927:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828500.673934:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828500.673969:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790495872/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828500.673997:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828500.674002:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828500.674007:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828500.674012:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5137 00000100:00080000:3.0:1787828500.674015:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828500.674019:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828500.674027:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828500.674038:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790496000/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828500.674047:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828500.674052:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828500.674053:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790495872:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828500.674053:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828500.674061:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5137 00000100:00080000:3.0:1787828500.674064:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828500.674066:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828500.674069:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828500.674076:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790496128/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828500.674083:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828500.674086:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828500.674087:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828500.674090:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5137 00000100:00080000:3.0:1787828500.674092:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828500.674094:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828500.674097:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828500.674104:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790496256/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828500.674111:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828500.674115:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828500.674116:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828500.674120:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5137 00000100:00080000:3.0:1787828500.674125:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828500.674141:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828500.674147:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828500.674162:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790496384/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828500.674177:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790496000:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828500.674179:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828500.674184:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828500.674186:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828500.674191:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5137 00000100:00080000:3.0:1787828500.674194:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828500.674198:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828500.674203:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828500.674213:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790496512/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828500.674220:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828500.674224:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790496128:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828500.674225:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828500.674229:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828500.674236:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5137 00000100:00080000:3.0:1787828500.674239:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828500.674241:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828500.674246:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828500.674257:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790496256:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828500.674258:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790496640/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828500.674269:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828500.674272:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828500.674274:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828500.674278:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5137 00000100:00080000:3.0:1787828500.674281:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828500.674285:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790496384:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787828500.674286:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828500.674295:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828500.674306:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790496768/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828500.674312:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790496512:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828500.674317:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828500.674322:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828500.674324:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828500.674328:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5137 00000100:00080000:3.0:1787828500.674331:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828500.674335:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790496640:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828500.674339:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828500.674346:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828500.674357:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790496768:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828500.674363:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790496896/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828500.674385:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790496896:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828501.185782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828501.377866:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828502.209811:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828502.401847:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828503.233849:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828503.425989:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787828503.746153:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:3.0:1787828503.746198:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787828503.746211:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787828503.746265:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:3.0:1787828503.746274:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787828503.746292:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787828503.746300:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787828503.746320:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787828503.746322:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828500/real 1787828503] req@ffff9e680a2d61c0 x1874668790496896/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787828503.746333:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787828503.746349:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787828503.746363:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790496896/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787828503.746365:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00080000:2.0:1787828503.746383:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828503.746391:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828503.746396:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790496896:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828503.746405:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828500/real 1787828503] req@ffff9e680a2d4b40 x1874668790496768/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828503.746419:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790496768/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828503.746425:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828503.746429:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828503.746432:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790496768:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828503.746439:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828500/real 1787828503] req@ffff9e680a2d7480 x1874668790496384/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828503.746451:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790496384/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828503.746460:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828503.746463:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828503.746466:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790496384:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828503.746473:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828500/real 1787828503] req@ffff9e680a2d5e00 x1874668790496512/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828503.746486:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790496512/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828503.746495:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828503.746498:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828503.746501:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790496512:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828503.746507:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828500/real 1787828503] req@ffff9e680a2d4f00 x1874668790496640/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828503.746517:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790496640/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828503.746524:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828503.746528:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828503.746531:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790496640:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828503.746537:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828500/real 1787828503] req@ffff9e680a2d52c0 x1874668790496000/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828503.746550:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790496000/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828503.746560:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828503.746564:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828503.746567:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790496000:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828503.746574:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828500/real 1787828503] req@ffff9e680a2d6d00 x1874668790496128/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828503.746586:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790496128/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828503.746593:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828503.746603:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828503.746607:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790496128:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828503.746614:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828500/real 1787828503] req@ffff9e680a2d7c00 x1874668790496256/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828503.746626:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790496256/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828503.746635:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828503.746639:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828503.746642:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790496256:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828503.746649:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828500/real 1787828503] req@ffff9e680a2d4780 x1874668790495872/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828503.746670:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790495872/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828503.746680:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828503.746683:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828503.746687:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790495872:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828504.257817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828504.450026:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828505.281816:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828505.473887:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828505.793791:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828505.793800:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828505.793803:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828505.793813:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5142 00000100:00080000:3.0:1787828505.793825:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828505.793833:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828505.793870:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790497024/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828505.793895:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828505.793898:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828505.793899:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828505.793902:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5142 00000100:00080000:3.0:1787828505.793905:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828505.793908:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828505.793916:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828505.793931:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790497152/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828505.793940:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790497024:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828505.793957:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828505.793962:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828505.793964:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828505.793969:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5142 00000100:00080000:3.0:1787828505.793975:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828505.793980:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828505.793987:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828505.793999:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828505.794005:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790497280/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828505.794018:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828505.794020:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828505.794022:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828505.794025:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5142 00000100:00080000:3.0:1787828505.794030:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828505.794033:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828505.794037:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828505/real 1787828505] req@ffff9e680a2d70c0 x1874668790497024/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828505.794040:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828505.794056:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790497408/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828505.794060:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790497024/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828505.794073:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828505.794079:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828505.794080:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828505.794082:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828505.794085:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828505.794109:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790497024:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828505.794109:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5142 00000100:00080000:3.0:1787828505.794119:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828505.794127:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828505.794133:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828505.794139:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790497152:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828505.794150:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790497536/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828505.794161:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828505.794167:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828505.794176:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828505.794181:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828505.794188:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5142 00000100:00080000:3.0:1787828505.794193:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828505.794194:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828505/real 1787828505] req@ffff9e680a2d4000 x1874668790497152/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828505.794203:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828505.794213:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828505.794222:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790497152/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828505.794235:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828505.794238:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790497664/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828505.794241:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828505.794250:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790497152:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828505.794262:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790497280:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828505.794265:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828505.794271:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828505.794274:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828505.794279:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5142 00000100:00080000:3.0:1787828505.794292:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828505.794294:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828505.794301:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828505.794308:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828505.794314:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828505/real 1787828505] req@ffff9e680a2d6580 x1874668790497280/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828505.794326:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790497792/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828505.794337:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828505.794340:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790497280/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828505.794342:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828505.794347:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828505.794361:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828505.794362:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5142 00000100:00080000:2.0:1787828505.794370:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828505.794371:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828505.794378:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790497280:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828505.794381:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828505.794390:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828505.794392:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790497408:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828505.794410:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790497920/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828505.794421:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828505.794428:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828505.794429:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828505.794434:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828505.794440:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5142 00000100:00080000:3.0:1787828505.794445:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828505.794450:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828505/real 1787828505] req@ffff9e680a2d43c0 x1874668790497408/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828505.794452:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828505.794461:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828505.794470:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790497408/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828505.794477:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790498048/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828505.794488:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828505.794497:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828505.794502:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790497408:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828505.794526:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790497536:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828505.794552:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828505.794565:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828505/real 1787828505] req@ffff9e680a2d5a40 x1874668790497536/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828505.794578:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790497536/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828505.794587:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828505.794591:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828505.794595:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790497536:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828505.794602:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790497664:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828505.794625:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828505.794639:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828505/real 1787828505] req@ffff9e680a2d70c0 x1874668790497664/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828505.794652:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790497664/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828505.794669:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828505.794673:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828505.794678:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790497664:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828505.794684:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790497792:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828505.794749:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828505.794764:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828505/real 1787828505] req@ffff9e680a2d4780 x1874668790497792/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828505.794777:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790497792/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828505.794791:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828505.794794:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828505.794798:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790497792:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828505.794804:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790497920:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828505.794825:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828505.794838:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828505/real 1787828505] req@ffff9e680a2d7c00 x1874668790497920/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828505.794851:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790497920/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828505.794877:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828505.794882:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828505.794886:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790497920:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828505.794893:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790498048:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828505.794918:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828505.794949:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828505/real 1787828505] req@ffff9e680a2d6d00 x1874668790498048/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828505.794964:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790498048/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828505.794972:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828505.794976:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828505.794980:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790498048:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828506.305783:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828506.497855:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828507.329760:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828507.521890:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828508.353815:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828508.545904:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828509.377786:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828509.570049:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828510.401846:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828510.593880:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828510.913899:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828510.913916:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828510.913923:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828510.913939:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5147 00000100:00080000:2.0:1787828510.913966:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828510.913980:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828510.914042:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790498176/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828510.914119:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828510.914124:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828510.914127:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828510.914132:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5147 00000100:00080000:2.0:1787828510.914139:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828510.914146:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828510.914159:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828510.914173:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790498176:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787828510.914183:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790498304/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828510.914200:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828510.914205:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828510.914207:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828510.914211:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5147 00000100:00080000:2.0:1787828510.914216:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828510.914222:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828510.914228:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828510.914243:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790498432/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828510.914256:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828510.914261:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828510.914264:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828510.914269:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5147 00000100:00080000:2.0:1787828510.914278:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787828510.914284:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828510.914294:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828510.914306:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828510.914331:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790498560/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828510.914349:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828510/real 1787828510] req@ffff9e6804423480 x1874668790498176/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828510.914358:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828510.914377:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828510.914384:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828510.914395:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790498176/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828510.914397:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5147 00000100:00080000:2.0:1787828510.914411:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828510.914422:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:3.0:1787828510.914423:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787828510.914436:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787828510.914437:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828510.914458:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790498176:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787828510.914475:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790498688/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828510.914490:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828510.914496:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1787828510.914501:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790498304:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828510.914502:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828510.914527:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5147 00000100:00080000:2.0:1787828510.914535:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828510.914540:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828510.914548:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828510.914569:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828510.914570:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790498816/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828510.914591:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828510.914599:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828510/real 1787828510] req@ffff9e68044212c0 x1874668790498304/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828510.914600:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828510.914604:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828510.914610:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5147 00000100:00080000:2.0:1787828510.914629:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828510.914641:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828510.914653:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828510.914666:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790498304/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828510.914670:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790498944/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828510.914737:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828510.914749:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828510.914766:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790498304:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828510.914777:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828510.914787:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787828510.914792:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790498432:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828510.914795:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828510.914812:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5147 00000100:00080000:2.0:1787828510.914828:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828510.914834:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828510.914844:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787828510.914866:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828510.914868:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790499072/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828510.914897:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828510.914906:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787828510.914906:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828510/real 1787828510] req@ffff9e68044230c0 x1874668790498432/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828510.914913:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828510.914924:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5147 00000100:00080000:2.0:1787828510.914932:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828510.914953:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828510.914967:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828510.914990:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790498432/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828510.914991:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790499200/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828510.915085:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828510.915093:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828510.915102:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790498432:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828510.915115:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790498560:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828510.915152:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828510.915168:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828510/real 1787828510] req@ffff9e68044221c0 x1874668790498560/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828510.915192:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790498560/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828510.915203:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828510.915207:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828510.915211:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790498560:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828510.915218:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790498688:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828510.915238:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828510.915249:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828510/real 1787828510] req@ffff9e6804423840 x1874668790498688/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828510.915263:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790498688/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828510.915273:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828510.915276:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828510.915280:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790498688:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828510.915303:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790498816:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828510.915324:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828510.915336:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828510/real 1787828510] req@ffff9e6804420b40 x1874668790498816/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828510.915349:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790498816/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828510.915369:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828510.915373:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828510.915377:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790498816:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828510.915384:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790498944:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828510.915403:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828510.915414:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828510/real 1787828510] req@ffff9e6804422580 x1874668790498944/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828510.915427:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790498944/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828510.915446:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828510.915449:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828510.915453:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790498944:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828510.915459:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790499072:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828510.915477:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828510.915487:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828510/real 1787828510] req@ffff9e6804421680 x1874668790499072/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828510.915500:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790499072/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828510.915517:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828510.915520:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828510.915525:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790499072:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828510.915532:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790499200:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828510.915550:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828510.915561:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828510/real 1787828510] req@ffff9e6804420780 x1874668790499200/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828510.915575:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790499200/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828535 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828510.915584:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828510.915588:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828510.915591:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790499200:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828511.425789:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828511.617979:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828512.449821:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828512.641943:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828513.473899:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828513.665962:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828514.497840:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828514.689919:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828515.521808:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828515.714082:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828516.033936:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828516.033959:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828516.033965:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828516.033978:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5152 00000100:00080000:0.0:1787828516.033992:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828516.034003:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828516.034082:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790499328/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828516.034116:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828516.034121:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828516.034124:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828516.034130:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5152 00000100:00080000:0.0:1787828516.034136:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828516.034143:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828516.034157:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828516.034187:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790499456/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828516.034189:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790499328:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828516.034215:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828516.034220:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828516.034223:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828516.034227:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5152 00000100:00080000:0.0:1787828516.034233:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828516.034238:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828516.034245:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828516.034264:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790499584/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828516.034276:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787828516.034282:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828516.034285:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828516.034290:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828516.034297:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5152 00000100:00080000:0.0:1787828516.034303:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828516.034308:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828516.034314:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828516.034328:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790499712/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828516.034350:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828516.034354:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828516.034356:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828516.034361:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5152 00000100:00080000:0.0:1787828516.034365:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828516.034369:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828516.034375:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828516.034389:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790499840/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828516.034400:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828516.034404:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828516.034407:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828516.034414:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5152 00000100:00080000:0.0:1787828516.034431:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828516.034442:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828516.034452:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828516.034469:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828516/real 1787828516] req@ffff9e6803e9a1c0 x1874668790499328/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828516.034483:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790499968/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828516.034503:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828516.034510:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828516.034514:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828516.034522:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5152 00000100:00080000:0.0:1787828516.034530:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828516.034536:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828516.034545:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828516.034573:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790500096/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828516.034593:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828516.034599:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828516.034603:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828516.034610:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5152 00000100:00080000:0.0:1787828516.034618:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828516.034626:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828516.034635:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828516.034666:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790500224/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828516.034687:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790499328/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828516.034690:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828516.034767:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828516.034770:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828516.034778:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5152 00000100:00080000:0.0:1787828516.034786:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828516.034793:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828516.034804:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828516.034830:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790500352/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828516.034844:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828516.034860:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828516.034872:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790499328:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828516.034935:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790499456:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828516.034975:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828516.035003:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828516/real 1787828516] req@ffff9e6803e99680 x1874668790499456/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828516.035019:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790499456/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828516.035063:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828516.035068:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828516.035072:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790499456:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828516.035080:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790499584:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828516.035107:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828516.035121:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828516/real 1787828516] req@ffff9e6803e9a580 x1874668790499584/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828516.035134:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790499584/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828516.035144:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828516.035148:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828516.035151:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790499584:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828516.035158:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790499712:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828516.035175:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828516.035185:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828516/real 1787828516] req@ffff9e6803e9b480 x1874668790499712/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828516.035198:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790499712/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828516.035208:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828516.035211:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828516.035214:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790499712:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828516.035220:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790499840:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828516.035237:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828516.035247:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828516/real 1787828516] req@ffff9e6803e9ad00 x1874668790499840/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828516.035260:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790499840/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828516.035269:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828516.035272:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828516.035276:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790499840:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828516.035300:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790499968:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828516.035321:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828516.035340:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828516/real 1787828516] req@ffff9e6803e992c0 x1874668790499968/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828516.035354:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790499968/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828516.035374:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828516.035378:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828516.035381:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790499968:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828516.035387:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790500096:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828516.035405:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828516.035416:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828516/real 1787828516] req@ffff9e6803e98f00 x1874668790500096/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828516.035429:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790500096/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828516.035438:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828516.035442:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828516.035446:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790500096:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828516.035452:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790500224:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828516.035470:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828516.035480:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828516/real 1787828516] req@ffff9e6803e99e00 x1874668790500224/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828516.035493:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790500224/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828516.035503:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828516.035506:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828516.035511:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790500224:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828516.035517:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790500352:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828516.035536:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828516.035547:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828516/real 1787828516] req@ffff9e6803e9a940 x1874668790500352/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828516.035560:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790500352/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828516.035568:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828516.035572:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828516.035575:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790500352:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828516.545867:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828516.738090:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828517.569843:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828517.762116:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828518.593833:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828518.785917:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828519.617811:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828519.809925:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828520.641925:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828520.834027:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828521.153786:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828521.153795:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828521.153799:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828521.153807:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5157 00000100:00080000:2.0:1787828521.153816:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828521.153822:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828521.153854:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790500480/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828521.153878:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828521.153880:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828521.153882:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828521.153884:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5157 00000100:00080000:2.0:1787828521.153887:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828521.153890:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828521.153898:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828521.153907:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790500608/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828521.153915:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828521.153920:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787828521.153921:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790500480:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828521.153923:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828521.153930:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5157 00000100:00080000:2.0:1787828521.153933:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828521.153935:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828521.153938:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828521.153946:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790500736/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828521.153953:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828521.153955:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828521.153956:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828521.153959:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5157 00000100:00080000:2.0:1787828521.153969:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828521.153972:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828521.153975:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787828521.153981:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828521.153997:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790500864/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828521.154011:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828521.154013:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828521.154014:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828521.154017:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5157 00000100:00080000:2.0:1787828521.154020:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828521.154023:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828521.154027:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828521.154033:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828521/real 1787828521] req@ffff9e6804422d00 x1874668790500480/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828521.154039:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790500992/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828521.154055:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828521.154057:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828521.154059:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828521.154061:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790500480/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828521.154064:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5157 00000100:00080000:2.0:1787828521.154070:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828521.154076:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828521.154082:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787828521.154084:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828521.154089:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828521.154092:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790501120/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828521.154096:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790500480:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828521.154112:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828521.154115:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828521.154116:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828521.154120:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5157 00000100:00080000:2.0:1787828521.154122:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828521.154124:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828521.154127:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828521.154131:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790500608:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828521.154137:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790501248/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828521.154144:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828521.154147:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828521.154148:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828521.154151:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5157 00000100:00080000:2.0:1787828521.154154:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787828521.154158:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828521.154160:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828521.154164:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828521.154179:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828521/real 1787828521] req@ffff9e6804422940 x1874668790500608/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828521.154180:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790501376/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828521.154212:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828521.154215:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787828521.154218:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790500608/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828521.154219:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828521.154226:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5157 00000100:00080000:2.0:1787828521.154232:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828521.154237:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:3.0:1787828521.154239:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787828521.154241:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787828521.154245:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828521.154249:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790500608:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828521.154252:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790501504/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828521.154259:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790500736:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828521.154284:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828521.154293:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828521/real 1787828521] req@ffff9e6804421e00 x1874668790500736/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828521.154300:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790500736/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828521.154305:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828521.154307:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828521.154309:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790500736:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828521.154313:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790500864:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828521.154323:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828521.154328:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828521/real 1787828521] req@ffff9e6804420f00 x1874668790500864/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828521.154348:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790500864/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828521.154357:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828521.154360:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828521.154363:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790500864:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828521.154369:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790500992:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828521.154390:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828521.154401:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828521/real 1787828521] req@ffff9e6804420780 x1874668790500992/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828521.154412:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790500992/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828521.154418:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828521.154420:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828521.154422:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790500992:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828521.154426:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790501120:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828521.154440:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828521.154447:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828521/real 1787828521] req@ffff9e6804421680 x1874668790501120/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828521.154454:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790501120/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828521.154464:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828521.154466:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828521.154468:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790501120:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828521.154482:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790501248:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828521.154493:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828521.154499:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828521/real 1787828521] req@ffff9e6804422580 x1874668790501248/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828521.154506:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790501248/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828521.154511:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828521.154513:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828521.154514:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790501248:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828521.154518:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790501376:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828521.154528:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828521.154533:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828521/real 1787828521] req@ffff9e6804420b40 x1874668790501376/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828521.154541:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790501376/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828521.154545:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828521.154547:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828521.154549:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790501376:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828521.154553:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790501504:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828521.154563:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828521.154568:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828521/real 1787828521] req@ffff9e6804423840 x1874668790501504/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828521.154575:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790501504/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828521.154579:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828521.154581:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828521.154583:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790501504:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828521.665835:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828521.857947:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828522.689819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828522.881874:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828523.713823:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828523.906287:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828524.737836:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828524.929906:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828525.761821:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828525.953849:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828526.273870:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828526.273883:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828526.273889:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828526.273901:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5162 00000100:00080000:1.0:1787828526.273915:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828526.273924:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828526.274002:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790501632/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828526.274039:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828526.274045:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828526.274047:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828526.274053:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5162 00000100:00080000:1.0:1787828526.274059:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828526.274066:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828526.274079:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828526.274096:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790501632:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828526.274103:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790501760/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828526.274121:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828526.274127:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828526.274130:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828526.274135:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5162 00000100:00080000:1.0:1787828526.274139:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828526.274144:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828526.274150:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828526.274164:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790501888/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828526.274178:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828526.274183:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828526.274185:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828526.274189:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5162 00000100:00080000:1.0:1787828526.274195:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787828526.274202:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828526.274209:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828526.274218:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828526.274235:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790502016/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828526.274247:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828526.274251:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828526.274254:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828526.274260:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5162 00000100:00080000:1.0:1787828526.274266:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828526.274271:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828526.274278:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828526/real 1787828526] req@ffff9e6831aabc00 x1874668790501632/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828526.274281:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828526.274314:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790502144/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828526.274321:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790501632/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828526.274356:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828526.274361:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828526.274373:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790501632:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828526.274389:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828526.274398:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828526.274403:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828526.274415:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5162 00000100:00100000:3.0:1787828526.274418:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790501760:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828526.274431:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828526.274445:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828526.274459:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828526.274464:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828526.274487:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828526/real 1787828526] req@ffff9e6831aab480 x1874668790501760/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828526.274499:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790502272/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828526.274515:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790501760/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828526.274542:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828526.274543:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828526.274550:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828526.274561:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790501760:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828526.274570:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790501888:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828526.274576:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828526.274584:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828526.274597:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5162 00000800:00000100:3.0:1787828526.274599:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828526.274608:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828526.274621:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828526.274624:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828526/real 1787828526] req@ffff9e6831aa9a40 x1874668790501888/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828526.274632:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828526.274657:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790502400/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828526.274663:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790501888/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828526.274737:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828526.274743:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828526.274749:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790501888:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828526.274761:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828526.274764:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790502016:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787828526.274780:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828526.274788:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828526.274802:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5162 00000800:00000100:3.0:1787828526.274812:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828526.274817:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828526.274834:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828526.274836:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828526/real 1787828526] req@ffff9e6831aa8000 x1874668790502016/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828526.274852:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828526.274861:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790502016/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828526.274875:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828526.274882:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790502528/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828526.274886:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828526.274900:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790502016:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787828526.274922:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828526.274924:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790502144:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787828526.274933:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828526.274939:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828526.274948:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5162 00000100:00080000:1.0:1787828526.274981:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828526.274988:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:3.0:1787828526.274996:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1787828526.275002:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828526.275020:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828526/real 1787828526] req@ffff9e6831aaad00 x1874668790502144/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828526.275028:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790502656/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828526.275064:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790502144/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828526.275077:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828526.275083:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828526.275089:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790502144:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828526.275115:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790502272:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828526.275138:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828526.275151:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828526/real 1787828526] req@ffff9e6831aaa1c0 x1874668790502272/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828526.275165:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790502272/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828526.275186:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828526.275190:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828526.275194:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790502272:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828526.275201:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790502400:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828526.275219:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828526.275230:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828526/real 1787828526] req@ffff9e6831aa8780 x1874668790502400/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828526.275243:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790502400/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828526.275252:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828526.275256:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828526.275260:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790502400:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828526.275267:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790502528:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828526.275284:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828526.275294:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828526/real 1787828526] req@ffff9e6831aaa940 x1874668790502528/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828526.275308:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790502528/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828526.275318:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828526.275321:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828526.275326:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790502528:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828526.275332:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790502656:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828526.275351:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828526.275361:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828526/real 1787828526] req@ffff9e6831aa8b40 x1874668790502656/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828526.275374:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790502656/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828526.275383:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828526.275387:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828526.275391:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790502656:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828526.785854:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828526.977996:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828527.809866:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828528.001962:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828528.833819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828529.025914:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828529.857824:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828530.049837:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828530.881808:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828531.073882:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828531.393839:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828531.393848:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828531.393853:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828531.393862:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5168 00000100:00080000:0.0:1787828531.393872:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828531.393879:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828531.393955:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790502784/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828531.393991:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828531.393994:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828531.393996:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828531.393999:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5168 00000100:00080000:0.0:1787828531.394019:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828531.394023:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828531.394034:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828531.394051:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790502912/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828531.394051:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790502784:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828531.394070:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828531.394073:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828531.394074:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828531.394078:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5168 00000100:00080000:0.0:1787828531.394082:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828531.394084:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828531.394088:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828531.394099:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790503040/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828531.394114:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828531.394118:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828531.394119:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828531.394124:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5168 00000800:00000100:3.0:1787828531.394129:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828531.394132:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828531.394141:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828531.394148:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828531.394165:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790503168/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828531.394176:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828531/real 1787828531] req@ffff9e6803e9b840 x1874668790502784/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828531.394185:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828531.394194:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828531.394197:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828531.394202:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790502784/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828531.394223:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5168 00000100:00080000:0.0:1787828531.394232:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828531.394248:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:3.0:1787828531.394249:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787828531.394255:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787828531.394256:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828531.394265:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790502784:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787828531.394276:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790503296/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828531.394302:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828531.394306:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828531.394308:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828531.394312:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790502912:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828531.394317:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5168 00000100:00080000:0.0:1787828531.394323:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828531.394327:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828531.394333:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828531.394344:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790503424/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828531.394353:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828531.394359:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828531.394363:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828531.394365:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828531.394371:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5168 00000100:00080000:0.0:1787828531.394374:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828531.394378:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828531.394385:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828531.394385:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828531/real 1787828531] req@ffff9e6803e9bc00 x1874668790502912/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828531.394400:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790503552/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828531.394414:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828531.394419:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790502912/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828531.394420:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828531.394424:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828531.394434:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5168 00000100:00080000:0.0:1787828531.394441:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828531.394442:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828531.394446:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:3.0:1787828531.394468:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:0.0:1787828531.394470:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828531.394475:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790502912:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828531.394483:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790503680/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828531.394486:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790503040:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828531.394508:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828531.394511:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828531.394513:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828531.394519:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5168 00000800:00000100:3.0:1787828531.394519:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828531.394527:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828531.394532:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828531.394537:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828531.394541:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828531/real 1787828531] req@ffff9e6803e98780 x1874668790503040/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828531.394550:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790503808/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828531.394558:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790503040/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828531.394570:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828531.394574:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828531.394577:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790503040:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828531.394584:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790503168:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828531.394599:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828531.394607:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828531/real 1787828531] req@ffff9e6803e98b40 x1874668790503168/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828531.394616:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790503168/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828531.394623:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828531.394625:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828531.394627:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790503168:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828531.394632:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790503296:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828531.394643:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828531.394650:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828531/real 1787828531] req@ffff9e6803e9a940 x1874668790503296/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828531.394658:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790503296/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828531.394665:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828531.394667:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828531.394669:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790503296:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828531.394684:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790503424:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828531.394697:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828531.394704:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828531/real 1787828531] req@ffff9e6803e99e00 x1874668790503424/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828531.394713:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790503424/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828531.394726:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828531.394729:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828531.394731:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790503424:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828531.394736:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790503552:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828531.394772:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828531.394803:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828531/real 1787828531] req@ffff9e6803e98f00 x1874668790503552/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828531.394813:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790503552/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828531.394820:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828531.394823:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828531.394826:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790503552:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828531.394830:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790503680:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828531.394844:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828531.394851:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828531/real 1787828531] req@ffff9e6803e992c0 x1874668790503680/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828531.394861:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790503680/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828531.394867:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828531.394869:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828531.394872:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790503680:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828531.394877:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790503808:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828531.394889:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828531.394896:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828531/real 1787828531] req@ffff9e6803e9ad00 x1874668790503808/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828531.394905:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790503808/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828531.394911:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828531.394913:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828531.394934:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790503808:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828531.905846:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828532.097895:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828532.929837:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828533.122009:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828533.953835:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828534.145856:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828534.977848:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828535.169943:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828536.001847:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828536.193834:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828536.513772:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828536.513782:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828536.513787:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828536.513799:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5173 00000100:00080000:0.0:1787828536.513810:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828536.513817:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828536.513853:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790503936/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828536.513893:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828536.513896:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828536.513898:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828536.513902:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5173 00000100:00080000:0.0:1787828536.513905:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828536.513909:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828536.513919:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828536.513931:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790504064/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828536.513942:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790503936:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828536.513944:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828536.513952:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828536.513957:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828536.513962:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5173 00000100:00080000:0.0:1787828536.513964:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828536.513967:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828536.513971:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828536.513982:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790504192/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828536.513990:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828536.513993:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828536.513996:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828536.513998:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5173 00000100:00080000:0.0:1787828536.514002:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828536.514005:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828536.514008:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828536.514018:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790504320/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828536.514025:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828536.514030:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828536.514034:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787828536.514034:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828536.514041:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5173 00000100:00080000:0.0:1787828536.514045:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828536.514048:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828536.514053:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828536.514064:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790504448/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828536.514071:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828536.514074:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828536.514076:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828536.514080:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5173 00000100:00080000:0.0:1787828536.514084:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828536.514087:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828536.514091:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828536.514097:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828536/real 1787828536] req@ffff9e6803e9b480 x1874668790503936/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828536.514105:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790504576/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828536.514141:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828536.514145:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828536.514147:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828536.514151:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5173 00000100:00080000:0.0:1787828536.514155:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828536.514158:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790503936/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828536.514160:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828536.514167:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828536.514181:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790504704/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828536.514186:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828536.514191:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828536.514195:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828536.514199:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828536.514199:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787828536.514208:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5173 00000100:00080000:0.0:1787828536.514212:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828536.514214:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790503936:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828536.514218:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828536.514224:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828536.514234:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790504832/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828536.514242:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828536.514245:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828536.514246:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828536.514251:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5173 00000100:00080000:0.0:1787828536.514254:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828536.514260:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828536.514261:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790504064:192.168.203.109@tcp:38:kworker.0 00000008:00100000:0.0:1787828536.514264:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828536.514278:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790504960/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828536.514311:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828536.514332:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828536/real 1787828536] req@ffff9e6803e9a580 x1874668790504064/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828536.514350:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790504064/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828536.514363:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828536.514368:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828536.514373:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790504064:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828536.514382:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790504192:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828536.514409:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828536.514424:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828536/real 1787828536] req@ffff9e6803e99680 x1874668790504192/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828536.514440:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790504192/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828536.514450:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828536.514455:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828536.514459:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790504192:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828536.514468:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790504320:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828536.514495:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828536.514510:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828536/real 1787828536] req@ffff9e6803e9a1c0 x1874668790504320/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828536.514527:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790504320/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828536.514539:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828536.514543:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828536.514547:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790504320:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828536.514556:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790504448:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828536.514581:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828536.514596:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828536/real 1787828536] req@ffff9e6803e9ad00 x1874668790504448/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828536.514610:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790504448/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828536.514622:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828536.514627:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828536.514631:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790504448:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828536.514638:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790504576:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828536.514661:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828536.514675:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828536/real 1787828536] req@ffff9e6803e992c0 x1874668790504576/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828536.514690:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790504576/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828536.514740:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828536.514745:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828536.514749:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790504576:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828536.514758:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790504704:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828536.514785:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828536.514801:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828536/real 1787828536] req@ffff9e6803e98f00 x1874668790504704/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828536.514817:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790504704/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828536.514828:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828536.514832:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828536.514837:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790504704:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828536.514845:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790504832:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828536.514870:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828536.514901:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828536/real 1787828536] req@ffff9e6803e99e00 x1874668790504832/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828536.514918:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790504832/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828536.514930:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828536.514934:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828536.514939:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790504832:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828536.514970:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790504960:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828536.514997:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828536.515012:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828536/real 1787828536] req@ffff9e6803e9a940 x1874668790504960/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828536.515029:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790504960/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828536.515040:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828536.515045:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828536.515049:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790504960:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828537.025849:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828537.217949:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828538.049842:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828538.241933:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828539.073790:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828539.265829:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828540.097788:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828540.289890:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828541.121830:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828541.313937:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828541.633925:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828541.633941:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828541.633948:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828541.633961:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5178 00000100:00080000:1.0:1787828541.633974:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828541.633983:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828541.634053:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790505088/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828541.634125:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828541.634133:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828541.634136:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828541.634145:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5178 00000100:00080000:1.0:1787828541.634152:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828541.634161:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828541.634176:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828541.634191:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790505088:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828541.634216:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790505216/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828541.634241:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828541.634258:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828541.634260:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828541.634266:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5178 00000100:00080000:1.0:1787828541.634273:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828541.634278:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828541.634285:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828541.634302:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828541.634305:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790505344/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828541.634331:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828541.634336:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828541.634338:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828541.634342:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5178 00000100:00080000:1.0:1787828541.634345:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828541.634349:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828541.634355:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828541.634368:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790505472/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828541.634380:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828541.634384:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828541.634402:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828541.634407:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5178 00000100:00080000:1.0:1787828541.634412:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828541.634417:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828541.634424:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828541.634426:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828541/real 1787828541] req@ffff9e6831aa9680 x1874668790505088/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828541.634442:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790505600/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828541.634461:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828541.634464:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828541.634467:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828541.634473:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5178 00000100:00100000:3.0:1787828541.634476:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790505088/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828541.634477:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828541.634486:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828541.634498:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787828541.634511:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828541.634522:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790505728/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828541.634525:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828541.634539:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790505088:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828541.634555:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828541.634563:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828541.634566:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828541.634572:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5178 00000100:00080000:1.0:1787828541.634578:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828541.634583:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828541.634590:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790505216:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787828541.634593:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828541.634613:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790505856/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828541.634626:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828541.634630:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828541.634633:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828541.634638:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5178 00000100:00080000:1.0:1787828541.634644:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787828541.634650:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828541.634655:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828541.634664:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828541.634679:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828541/real 1787828541] req@ffff9e6831aaa580 x1874668790505216/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828541.634681:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790505984/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828541.634750:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828541.634765:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828541.634771:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828541.634782:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5178 00000100:00100000:3.0:1787828541.634784:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790505216/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828541.634792:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828541.634806:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:3.0:1787828541.634814:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787828541.634816:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787828541.634830:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828541.634839:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790506112/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828541.634862:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790505216:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828541.634888:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790505344:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828541.634966:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828541.634989:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828541/real 1787828541] req@ffff9e6831aa8f00 x1874668790505344/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828541.635006:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790505344/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828541.635016:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828541.635020:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828541.635024:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790505344:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828541.635032:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790505472:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828541.635057:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828541.635069:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828541/real 1787828541] req@ffff9e6831aa9e00 x1874668790505472/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828541.635083:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790505472/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828541.635101:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828541.635104:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828541.635108:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790505472:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828541.635114:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790505600:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828541.635132:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828541.635142:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828541/real 1787828541] req@ffff9e6831aa8b40 x1874668790505600/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828541.635155:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790505600/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828541.635165:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828541.635168:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828541.635171:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790505600:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828541.635178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790505728:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828541.635212:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828541.635222:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828541/real 1787828541] req@ffff9e6831aaa940 x1874668790505728/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828541.635233:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790505728/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828541.635250:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828541.635253:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828541.635257:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790505728:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828541.635278:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790505856:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828541.635315:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828541.635328:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828541/real 1787828541] req@ffff9e6831aa8780 x1874668790505856/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828541.635351:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790505856/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828541.635361:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828541.635364:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828541.635367:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790505856:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828541.635374:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790505984:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828541.635392:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828541.635402:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828541/real 1787828541] req@ffff9e6831aaa1c0 x1874668790505984/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828541.635415:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790505984/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828541.635424:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828541.635427:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828541.635431:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790505984:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828541.635437:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790506112:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828541.635454:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828541.635464:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828541/real 1787828541] req@ffff9e6831aaad00 x1874668790506112/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828541.635477:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790506112/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828541.635485:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828541.635488:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828541.635491:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790506112:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828542.145838:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828542.337900:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828543.169828:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828543.361998:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828544.193851:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828544.385927:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828545.217884:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828545.409880:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828546.241876:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828546.433885:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828546.753990:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828546.754008:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828546.754016:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828546.754035:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5183 00000100:00080000:1.0:1787828546.754053:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828546.754068:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828546.754149:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790506240/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828546.754199:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828546.754208:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828546.754212:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828546.754221:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5183 00000100:00080000:1.0:1787828546.754229:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828546.754237:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828546.754255:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828546.754285:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790506368/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828546.754308:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828546.754315:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828546.754319:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828546.754327:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5183 00000100:00080000:1.0:1787828546.754335:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828546.754342:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828546.754352:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828546.754377:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790506496/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828546.754406:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828546.754430:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828546.754435:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828546.754444:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5183 00000100:00080000:1.0:1787828546.754461:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828546.754470:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828546.754481:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828546.754505:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790506624/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828546.754522:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828546.754528:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828546.754531:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828546.754537:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5183 00000100:00080000:1.0:1787828546.754543:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828546.754550:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828546.754558:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828546.754591:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790506752/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828546.754605:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828546.754610:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828546.754612:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828546.754619:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5183 00000100:00080000:1.0:1787828546.754624:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828546.754631:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828546.754638:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828546.754652:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790506880/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828546.754666:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828546.754671:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828546.754673:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828546.754679:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5183 00000100:00080000:1.0:1787828546.754684:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828546.754690:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828546.754698:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828546.754750:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790507008/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828546.754766:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828546.754771:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828546.754775:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828546.754781:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5183 00000100:00080000:1.0:1787828546.754787:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828546.754793:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828546.754800:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828546.754845:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790507136/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828546.754868:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828546.754873:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828546.754876:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828546.754881:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5183 00000100:00080000:1.0:1787828546.754886:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828546.754892:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828546.754898:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828546.754912:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790507264/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828546.755180:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790506240:192.168.203.109@tcp:250:kworker.0 00000800:00000100:3.0:1787828546.755271:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828546.755335:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828546/real 1787828546] req@ffff9e6831aa8000 x1874668790506240/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828546.755367:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790506240/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828546.755386:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828546.755391:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828546.755395:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790506240:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828546.755406:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790506368:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828546.755439:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828546.755455:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828546/real 1787828546] req@ffff9e6831aa9a40 x1874668790506368/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828546.755469:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790506368/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828546.755479:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828546.755483:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828546.755487:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790506368:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828546.755494:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790506496:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828546.755512:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828546.755522:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828546/real 1787828546] req@ffff9e6831aab480 x1874668790506496/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828546.755534:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790506496/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828546.755543:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828546.755546:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828546.755550:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790506496:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828546.755556:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790506624:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828546.755573:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828546.755582:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828546/real 1787828546] req@ffff9e6831aabc00 x1874668790506624/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828546.755595:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790506624/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828546.755604:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828546.755607:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828546.755610:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790506624:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828546.755616:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790506752:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828546.755642:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828546.755660:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828546/real 1787828546] req@ffff9e6831aaad00 x1874668790506752/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828546.755681:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790506752/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828546.755733:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828546.755753:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828546.755761:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790506752:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828546.755774:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790506880:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828546.755836:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828546.755856:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828546/real 1787828546] req@ffff9e6831aaa1c0 x1874668790506880/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828546.755869:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790506880/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828546.755888:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828546.755891:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828546.755895:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790506880:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828546.755902:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790507008:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828546.755924:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828546.755935:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828546/real 1787828546] req@ffff9e6831aa8780 x1874668790507008/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828546.755946:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790507008/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828546.755955:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828546.755958:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828546.755961:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790507008:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828546.755966:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790507136:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828546.755982:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828546.755992:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828546/real 1787828546] req@ffff9e6831aaa940 x1874668790507136/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828546.756003:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790507136/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828546.756011:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828546.756014:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828546.756018:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790507136:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828546.756024:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790507264:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828546.756040:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828546.756049:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828546/real 1787828546] req@ffff9e6831aa8b40 x1874668790507264/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828546.756067:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790507264/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828546.756075:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828546.756078:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828546.756081:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790507264:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828547.265899:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828547.457996:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828548.289871:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828548.481931:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828549.313807:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828549.505920:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828550.337768:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828550.529883:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828551.361895:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828551.554002:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828551.873908:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828551.873924:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828551.873931:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828551.873947:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5188 00000100:00080000:1.0:1787828551.873975:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828551.873990:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828551.874073:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790507392/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828551.874127:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828551.874136:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828551.874141:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828551.874152:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5188 00000100:00080000:1.0:1787828551.874163:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828551.874173:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828551.874190:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828551.874195:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790507392:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828551.874225:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790507520/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828551.874253:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828551.874261:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828551.874265:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828551.874275:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5188 00000100:00080000:1.0:1787828551.874320:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828551.874331:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828551.874333:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828551.874348:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828551.874390:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790507648/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828551.874410:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828551.874419:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787828551.874420:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828551/real 1787828551] req@ffff9e6831aa9e00 x1874668790507392/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828551.874423:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828551.874437:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5188 00000100:00080000:1.0:1787828551.874449:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828551.874459:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828551.874469:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828551.874471:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790507392/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828551.874496:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790507776/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828551.874503:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828551.874518:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828551.874522:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828551.874528:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790507392:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828551.874532:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828551.874541:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828551.874548:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5188 00000100:00080000:1.0:1787828551.874554:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828551.874570:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790507520:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828551.874574:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828551.874587:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828551.874612:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790507904/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828551.874622:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828551.874645:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828551.874648:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828551/real 1787828551] req@ffff9e6831aa8f00 x1874668790507520/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828551.874658:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828551.874667:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828551.874681:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5188 00000100:00100000:0.0:1787828551.874684:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790507520/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828551.874755:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828551.874756:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828551.874765:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828551.874817:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828551.874819:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790507520:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787828551.874836:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828551.874839:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790507648:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828551.874872:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790508032/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828551.874888:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828551.874895:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:0.0:1787828551.874896:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828551.874902:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828551.874912:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5188 00000100:00080000:1.0:1787828551.874919:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828551.874930:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828551.874938:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828551/real 1787828551] req@ffff9e6831aaa580 x1874668790507648/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828551.874944:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828551.874971:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790507648/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828551.874977:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790508160/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828551.875031:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828551.875040:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828551.875046:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828551.875054:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787828551.875058:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790507648:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828551.875060:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828551.875074:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5188 00000100:00100000:0.0:1787828551.875080:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790507776:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787828551.875084:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828551.875093:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828551.875102:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828551.875111:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828551.875125:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790508288/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828551.875132:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828551/real 1787828551] req@ffff9e6831aa9680 x1874668790507776/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828551.875154:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828551.875165:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828551.875168:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828551.875174:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790507776/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828551.875179:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5188 00000100:00080000:1.0:1787828551.875188:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828551.875199:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787828551.875203:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787828551.875207:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787828551.875213:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828551.875221:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790507776:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828551.875228:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790508416/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828551.875257:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790507904:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828551.875282:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828551.875293:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828551/real 1787828551] req@ffff9e6831aa9e00 x1874668790507904/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828551.875313:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790507904/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828551.875335:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828551.875340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828551.875347:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790507904:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828551.875358:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790508032:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828551.875392:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828551.875412:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828551/real 1787828551] req@ffff9e6831aa8b40 x1874668790508032/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828551.875427:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790508032/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828551.875438:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828551.875442:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828551.875446:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790508032:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828551.875452:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790508160:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828551.875476:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828551.875489:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828551/real 1787828551] req@ffff9e6831aaa940 x1874668790508160/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828551.875511:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790508160/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828551.875524:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828551.875529:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828551.875535:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790508160:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828551.875546:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790508288:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828551.875574:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828551.875588:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828551/real 1787828551] req@ffff9e6831aa8780 x1874668790508288/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828551.875602:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790508288/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828551.875611:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828551.875616:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828551.875622:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790508288:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828551.875632:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790508416:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828551.875656:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828551.875668:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828551/real 1787828551] req@ffff9e6831aaa1c0 x1874668790508416/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828551.875699:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790508416/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828576 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828551.875757:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828551.875762:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828551.875767:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790508416:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828552.385810:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828552.578078:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828553.409806:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828553.601856:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828554.433922:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828554.625956:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828555.457873:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828555.649971:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828556.481854:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828556.674204:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828556.993821:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828556.993832:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828556.993836:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828556.993845:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5193 00000100:00080000:0.0:1787828556.993855:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828556.993863:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828556.993900:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790508544/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828556.993925:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828556.993927:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828556.993929:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828556.993932:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5193 00000100:00080000:0.0:1787828556.993935:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828556.993939:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828556.993947:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828556.993956:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790508672/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828556.993964:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790508544:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828556.993968:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828556.993971:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828556.993973:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828556.993975:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5193 00000100:00080000:0.0:1787828556.993977:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828556.993979:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828556.993983:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828556.993991:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790508800/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828556.993998:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828556.994000:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828556.994001:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828556.994006:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5193 00000100:00080000:0.0:1787828556.994011:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828556.994016:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:3.0:1787828556.994024:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787828556.994028:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828556.994044:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790508928/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828556.994056:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828556.994060:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828556.994062:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828556.994068:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5193 00000100:00100000:3.0:1787828556.994069:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828556/real 1787828556] req@ffff9e6803e98b40 x1874668790508544/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828556.994072:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828556.994079:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828556.994087:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828556.994093:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790508544/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828556.994100:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790509056/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828556.994129:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828556.994130:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828556.994137:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828556.994138:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787828556.994140:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828556.994145:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790508544:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828556.994146:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5193 00000100:00080000:0.0:1787828556.994152:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828556.994156:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828556.994159:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828556.994168:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790509184/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828556.994172:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790508672:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828556.994188:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828556.994193:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828556.994196:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787828556.994197:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828556.994205:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5193 00000100:00080000:0.0:1787828556.994211:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828556.994213:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828556/real 1787828556] req@ffff9e6803e98780 x1874668790508672/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828556.994219:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828556.994228:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828556.994231:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790508672/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828556.994241:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828556.994243:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790509312/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828556.994246:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828556.994254:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790508672:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828556.994262:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790508800:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828556.994263:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828556.994269:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828556.994272:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828556.994279:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5193 00000800:00000100:3.0:1787828556.994279:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828556.994286:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828556.994293:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828556.994299:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828556.994300:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828556/real 1787828556] req@ffff9e6803e9bc00 x1874668790508800/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828556.994312:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790508800/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828556.994317:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790509440/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828556.994321:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828556.994330:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828556.994335:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790508800:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828556.994336:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828556.994343:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787828556.994345:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790508928:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787828556.994347:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828556.994355:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5193 00000100:00080000:0.0:1787828556.994360:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828556.994363:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828556.994368:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787828556.994368:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828556.994382:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790509568/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828556.994383:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828556/real 1787828556] req@ffff9e6803e9b840 x1874668790508928/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828556.994416:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790508928/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828556.994423:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828556.994427:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828556.994429:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790508928:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828556.994433:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790509056:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828556.994444:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828556.994450:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828556/real 1787828556] req@ffff9e6803e9a940 x1874668790509056/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828556.994470:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790509056/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828556.994479:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828556.994483:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828556.994487:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790509056:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828556.994506:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790509184:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828556.994527:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828556.994536:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828556/real 1787828556] req@ffff9e6803e99e00 x1874668790509184/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828556.994545:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790509184/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828556.994552:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828556.994554:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828556.994556:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790509184:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828556.994576:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790509312:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787828556.994590:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828556.994597:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828556/real 1787828556] req@ffff9e6803e98f00 x1874668790509312/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828556.994605:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790509312/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828556.994612:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828556.994615:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828556.994618:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790509312:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828556.994640:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790509440:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828556.994664:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828556.994677:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828556/real 1787828556] req@ffff9e6803e992c0 x1874668790509440/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828556.994707:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790509440/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828556.994719:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828556.994721:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828556.994724:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790509440:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828556.994728:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790509568:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828556.994776:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828556.994786:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828556/real 1787828556] req@ffff9e6803e9ad00 x1874668790509568/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828556.994794:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790509568/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828581 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828556.994800:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828556.994802:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828556.994804:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790509568:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828557.505875:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828557.698037:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828558.529931:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828558.721989:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828559.553954:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828559.745997:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828560.577871:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828560.769953:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828561.601832:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828561.793939:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828562.113810:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828562.113822:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828562.113826:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828562.113837:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5198 00000100:00080000:3.0:1787828562.113849:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828562.113857:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828562.113892:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790509696/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828562.113923:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828562.113927:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828562.113930:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828562.113935:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5198 00000100:00080000:3.0:1787828562.113939:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828562.113944:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828562.113966:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828562.113971:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790509696:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828562.113979:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790509824/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828562.113990:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828562.113993:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828562.113995:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828562.114000:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5198 00000100:00080000:3.0:1787828562.114004:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828562.114008:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828562.114013:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828562.114034:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790509952/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828562.114044:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828562.114048:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828562.114050:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828562.114054:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5198 00000800:00000100:0.0:1787828562.114055:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828562.114057:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828562.114061:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828562.114066:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828562.114076:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790510080/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828562.114085:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828562.114088:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828562.114090:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828562.114094:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5198 00000100:00100000:0.0:1787828562.114096:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828562/real 1787828562] req@ffff9e680a2d52c0 x1874668790509696/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828562.114098:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828562.114102:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828562.114107:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828562.114116:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790509696/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828562.114117:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790510208/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828562.114128:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828562.114129:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828562.114132:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828562.114134:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828562.114134:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828562.114139:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790509696:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828562.114139:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5198 00000100:00080000:3.0:1787828562.114143:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828562.114146:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828562.114150:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828562.114159:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790510336/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828562.114163:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790509824:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828562.114177:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828562.114181:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828562.114183:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828562.114187:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5198 00000800:00000100:0.0:1787828562.114188:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828562.114191:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828562.114195:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828562.114199:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828562.114213:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828562/real 1787828562] req@ffff9e680a2d4f00 x1874668790509824/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828562.114213:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790510464/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828562.114225:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828562.114229:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790509824/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828562.114229:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828562.114232:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828562.114235:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5198 00000100:00080000:0.0:1787828562.114240:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828562.114240:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828562.114244:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828562.114245:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828562.114248:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790509824:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787828562.114250:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828562.114256:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790509952:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828562.114260:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790510592/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828562.114270:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828562.114273:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828562.114275:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787828562.114276:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828562.114280:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5198 00000100:00080000:3.0:1787828562.114284:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828562.114288:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828562/real 1787828562] req@ffff9e680a2d5e00 x1874668790509952/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828562.114289:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828562.114294:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828562.114302:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790509952/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828562.114304:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790510720/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828562.114321:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828562.114325:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828562.114329:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790509952:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828562.114337:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790510080:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828562.114361:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828562.114372:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828562/real 1787828562] req@ffff9e680a2d7480 x1874668790510080/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828562.114383:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790510080/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828562.114393:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828562.114396:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828562.114400:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790510080:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828562.114407:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790510208:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828562.114430:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828562.114450:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828562/real 1787828562] req@ffff9e680a2d4b40 x1874668790510208/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828562.114460:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790510208/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828562.114466:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828562.114469:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828562.114471:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790510208:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828562.114487:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790510336:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828562.114513:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828562.114520:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828562/real 1787828562] req@ffff9e680a2d61c0 x1874668790510336/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828562.114529:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790510336/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828562.114536:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828562.114538:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828562.114540:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790510336:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828562.114544:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790510464:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828562.114554:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828562.114571:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828562/real 1787828562] req@ffff9e680a2d52c0 x1874668790510464/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828562.114579:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790510464/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828562.114585:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828562.114587:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828562.114589:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790510464:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828562.114593:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790510592:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828562.114603:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828562.114610:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828562/real 1787828562] req@ffff9e680a2d6d00 x1874668790510592/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828562.114639:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790510592/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828562.114658:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828562.114661:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828562.114664:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790510592:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828562.114671:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790510720:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828562.114691:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828562.114785:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828562/real 1787828562] req@ffff9e680a2d7c00 x1874668790510720/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828562.114796:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790510720/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828562.114802:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828562.114805:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828562.114807:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790510720:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828562.625883:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828562.817876:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828563.649836:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828563.841866:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828564.673795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828564.865928:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828565.697844:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828565.889928:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828566.721820:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:0.0:1787828566.914016:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828567.233810:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828567.233818:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828567.233821:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828567.233830:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5203 00000100:00080000:0.0:1787828567.233839:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828567.233844:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828567.233877:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790510848/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828567.233900:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828567.233903:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828567.233904:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828567.233907:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5203 00000100:00080000:0.0:1787828567.233909:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828567.233912:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828567.233920:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828567.233934:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790510976/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828567.233935:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790510848:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828567.233949:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828567.233952:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828567.233953:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828567.233955:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5203 00000100:00080000:0.0:1787828567.233958:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828567.233961:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828567.233964:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828567.233972:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790511104/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828567.233979:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828567.233982:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828567.233983:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828567.233985:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5203 00000100:00080000:0.0:1787828567.233990:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828567.233992:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828567.233996:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828567.234003:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790511232/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828567.234009:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828567.234011:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828567.234013:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828567.234027:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5203 00000100:00080000:0.0:1787828567.234030:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828567.234033:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828567.234036:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828567.234047:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790511360/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828567.234047:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790510976:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828567.234061:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828567.234064:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828567.234065:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828567.234069:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5203 00000100:00080000:0.0:1787828567.234072:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828567.234075:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828567.234079:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828567.234083:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790511104:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828567.234089:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790511488/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828567.234096:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828567.234099:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828567.234110:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828567.234114:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5203 00000100:00100000:3.0:1787828567.234117:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790511232:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787828567.234119:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828567.234126:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828567.234131:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828567.234141:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790511616/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828567.234147:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790511360:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787828567.234150:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828567.234154:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828567.234156:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828567.234159:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5203 00000100:00080000:0.0:1787828567.234163:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828567.234167:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828567.234171:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790511488:192.168.203.109@tcp:38:kworker.0 00000008:00100000:0.0:1787828567.234172:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828567.234187:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790511744/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828567.234197:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790511616:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828567.234212:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828567.234215:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828567.234217:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828567.234221:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5203 00000100:00100000:3.0:1787828567.234223:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790511744:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787828567.234225:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828567.234231:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828567.234235:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828567.234246:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790511872/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828567.234279:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790511872:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828567.745813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828567.937867:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828568.769826:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828568.961893:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828569.793875:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828569.985835:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787828570.306061:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:0.0:1787828570.306138:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787828570.306177:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828570.306254:0:2387:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:0.0:1787828570.306276:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828570.306305:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828570.306318:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828570.306325:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828570.306336:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828570.306345:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828570.306355:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:3.0:1787828570.306374:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828567/real 1787828570] req@ffff9e6803e9a940 x1874668790511872/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828570.306418:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790511872/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828570.306440:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828570.306449:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828570.306457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790511872:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828570.306470:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828567/real 1787828570] req@ffff9e6803e99e00 x1874668790511744/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828570.306487:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790511744/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828570.306501:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828570.306507:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828570.306513:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790511744:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828570.306525:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828567/real 1787828570] req@ffff9e6803e98f00 x1874668790511616/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828570.306545:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790511616/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828570.306559:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828570.306565:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828570.306571:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790511616:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828570.306583:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828567/real 1787828570] req@ffff9e6803e9ad00 x1874668790511360/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828570.306604:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790511360/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828570.306619:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828570.306633:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828570.306639:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790511360:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828570.306679:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828567/real 1787828570] req@ffff9e6803e992c0 x1874668790511488/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828570.306752:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790511488/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828570.306770:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828570.306777:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828570.306783:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790511488:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828570.306794:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828567/real 1787828570] req@ffff9e6803e99680 x1874668790510976/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828570.306816:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790510976/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828570.306830:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828570.306836:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828570.306841:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790510976:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828570.306852:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828567/real 1787828570] req@ffff9e6803e9a580 x1874668790511104/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828570.306871:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790511104/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828570.306886:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828570.306893:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828570.306900:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790511104:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828570.306912:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828567/real 1787828570] req@ffff9e6803e9b480 x1874668790511232/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828570.306932:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790511232/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828570.307348:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828570.307356:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828570.307362:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790511232:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828570.307374:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828567/real 1787828570] req@ffff9e6803e9a1c0 x1874668790510848/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828570.307398:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790510848/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828570.307412:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828570.307417:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828570.307422:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790510848:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828570.817856:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828571.009871:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828571.841830:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828572.033988:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828572.353840:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828572.353855:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828572.353861:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828572.353891:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5209 00000100:00080000:3.0:1787828572.353906:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828572.353916:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828572.353961:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790512000/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828572.354008:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828572.354012:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828572.354013:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828572.354017:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5209 00000100:00080000:3.0:1787828572.354020:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828572.354023:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828572.354034:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828572.354045:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790512128/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828572.354054:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828572.354058:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828572.354060:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828572.354066:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790512000:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828572.354068:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5209 00000100:00080000:3.0:1787828572.354076:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828572.354090:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828572.354098:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828572.354115:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790512256/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828572.354134:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828572.354138:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828572.354140:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828572.354146:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5209 00000100:00080000:3.0:1787828572.354151:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828572.354157:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828572.354159:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828572.354169:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828572.354186:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790512384/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828572.354198:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828572.354202:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828572.354206:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828572.354212:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5209 00000100:00080000:3.0:1787828572.354217:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828572.354222:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828572/real 1787828572] req@ffff9e680a2d4780 x1874668790512000/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828572.354229:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828572.354241:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828572.354258:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790512000/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828572.354261:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790512512/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828572.354286:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828572.354296:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828572.354298:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828572.354313:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828572.354316:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828572.354323:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5209 00000100:00100000:0.0:1787828572.354332:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790512000:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828572.354334:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828572.354348:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828572.354355:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828572.354373:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790512640/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828572.354376:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790512128:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828572.354413:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828572.354417:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828572.354420:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828572.354425:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5209 00000100:00080000:3.0:1787828572.354431:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828572.354438:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828572.354443:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828572.354452:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828572.354462:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828572/real 1787828572] req@ffff9e680a2d70c0 x1874668790512128/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828572.354474:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790512768/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828572.354488:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790512128/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828572.354498:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828572.354515:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828572.354524:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828572.354527:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828572.354534:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5209 00000100:00080000:0.0:1787828572.354543:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828572.354544:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828572.354563:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828572.354565:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790512128:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787828572.354578:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828572.354585:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790512256:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828572.354620:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828572.354652:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828572/real 1787828572] req@ffff9e680a2d5a40 x1874668790512256/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828572.354673:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790512256/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828572.354682:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828572.354686:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828572.354690:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790512256:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828572.354732:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790512384:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828572.354759:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828572.354777:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790512896/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828572.354778:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828572/real 1787828572] req@ffff9e680a2d43c0 x1874668790512384/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828572.354840:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790512384/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828572.354854:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828572.354858:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828572.354861:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828572.354869:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828572.354873:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828572.354880:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5209 00000100:00100000:0.0:1787828572.354886:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790512384:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787828572.354890:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828572.354899:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790512512:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787828572.354900:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828572.354913:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828572.354939:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828572.354939:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790513024/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828572.354957:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828572/real 1787828572] req@ffff9e680a2d6580 x1874668790512512/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828572.354972:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790512512/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828572.354989:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828572.354994:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828572.354998:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790512512:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828572.355021:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790512640:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828572.355040:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828572.355051:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828572/real 1787828572] req@ffff9e680a2d4000 x1874668790512640/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828572.355062:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790512640/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828572.355072:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828572.355075:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828572.355078:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790512640:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828572.355085:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790512768:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828572.355102:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828572.355113:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828572/real 1787828572] req@ffff9e680a2d4780 x1874668790512768/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828572.355123:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790512768/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828572.355131:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828572.355134:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828572.355138:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790512768:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828572.355144:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790512896:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828572.355161:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828572.355171:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828572/real 1787828572] req@ffff9e680a2d7c00 x1874668790512896/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828572.355182:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790512896/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828572.355189:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828572.355192:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828572.355196:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790512896:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828572.355202:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790513024:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828572.355219:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828572.355228:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828572/real 1787828572] req@ffff9e680a2d6d00 x1874668790513024/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828572.355239:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790513024/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828572.355247:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828572.355250:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828572.355253:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790513024:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828572.865828:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828573.057906:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828573.889904:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828574.081985:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828574.913906:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828575.105822:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828575.937818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828576.129941:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828576.961802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828577.153926:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828577.473799:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828577.473810:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828577.473816:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828577.473828:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5214 00000100:00080000:3.0:1787828577.473840:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828577.473850:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828577.473892:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790513152/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828577.473924:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828577.473929:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828577.473931:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828577.473936:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5214 00000100:00080000:3.0:1787828577.473942:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828577.473957:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828577.473970:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828577.473988:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790513280/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828577.473996:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790513152:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828577.474010:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828577.474016:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828577.474019:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828577.474025:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5214 00000100:00080000:3.0:1787828577.474030:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828577.474035:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828577.474065:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828577.474084:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828577.474086:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790513408/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828577.474101:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828577.474106:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828577.474108:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828577.474112:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5214 00000100:00080000:3.0:1787828577.474118:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828577.474123:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828577.474131:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828577.474139:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828577/real 1787828577] req@ffff9e680a2d52c0 x1874668790513152/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828577.474150:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790513536/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828577.474163:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790513152/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828577.474193:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828577.474194:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828577.474201:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828577.474204:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828577.474210:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828577.474216:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790513152:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828577.474221:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5214 00000100:00080000:3.0:1787828577.474233:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828577.474238:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828577.474245:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828577.474259:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790513280:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828577.474262:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790513664/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828577.474280:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828577.474286:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828577.474289:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828577.474293:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5214 00000100:00080000:3.0:1787828577.474299:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828577.474300:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828577.474311:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828577.474318:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828577.474321:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828577/real 1787828577] req@ffff9e680a2d61c0 x1874668790513280/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828577.474338:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790513792/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828577.474339:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790513280/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828577.474369:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828577.474378:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828577.474380:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828577.474384:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790513280:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828577.474391:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787828577.474397:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790513408:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828577.474397:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828577.474410:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5214 00000100:00080000:3.0:1787828577.474417:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828577.474425:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828577.474435:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828577.474445:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828577/real 1787828577] req@ffff9e680a2d4b40 x1874668790513408/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828577.474460:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828577.474462:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790513408/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828577.474480:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828577.474485:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790513920/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828577.474487:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828577.474497:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790513408:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828577.474511:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790513536:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787828577.474518:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828577.474524:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828577.474527:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787828577.474533:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828577.474539:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5214 00000100:00080000:3.0:1787828577.474547:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828577.474550:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828577/real 1787828577] req@ffff9e680a2d7480 x1874668790513536/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828577.474559:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828577.474567:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790513536/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828577.474569:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787828577.474579:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828577.474584:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828577.474587:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790513536:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828577.474613:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790514048/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828577.474629:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790513664:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787828577.474630:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828577.474651:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828577.474654:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828577.474660:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5214 00000100:00080000:3.0:1787828577.474666:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828577.474672:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787828577.474679:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787828577.474684:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828577.474732:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828577/real 1787828577] req@ffff9e680a2d5e00 x1874668790513664/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828577.474746:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790513664/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828577.474754:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790514176/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828577.474766:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828577.474780:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828577.474792:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790513664:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828577.474803:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790513792:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828577.474833:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828577.474848:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828577/real 1787828577] req@ffff9e680a2d4f00 x1874668790513792/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828577.474863:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790513792/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828577.474882:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828577.474886:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828577.474890:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790513792:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828577.474898:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790513920:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828577.474923:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828577.474935:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828577/real 1787828577] req@ffff9e680a2d52c0 x1874668790513920/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828577.474946:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790513920/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828577.474957:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828577.474960:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828577.474964:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790513920:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828577.474985:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790514048:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828577.475013:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828577.475027:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828577/real 1787828577] req@ffff9e680a2d6d00 x1874668790514048/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828577.475036:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790514048/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828577.475044:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828577.475046:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828577.475049:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790514048:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828577.475053:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790514176:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828577.475068:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828577.475076:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828577/real 1787828577] req@ffff9e680a2d7c00 x1874668790514176/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828577.475084:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790514176/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828577.475091:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828577.475093:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828577.475095:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790514176:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828577.985779:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828578.177886:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828579.009802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828579.201932:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828580.033817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828580.226025:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828581.057790:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828581.249845:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828582.081765:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828582.273924:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828582.593793:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828582.593802:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828582.593807:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828582.593817:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5219 00000100:00080000:3.0:1787828582.593827:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828582.593835:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828582.593871:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790514304/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828582.593901:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828582.593905:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828582.593907:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828582.593911:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5219 00000100:00080000:3.0:1787828582.593915:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828582.593920:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828582.593930:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828582.593945:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790514304:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828582.593946:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790514432/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828582.593964:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828582.593968:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828582.593970:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828582.593974:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5219 00000100:00080000:3.0:1787828582.593979:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828582.593984:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828582.593989:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828582.594001:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790514560/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828582.594011:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828582.594018:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828582.594022:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828582.594025:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828582.594029:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5219 00000100:00080000:3.0:1787828582.594034:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828582.594039:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828582.594044:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828582.594048:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828582/real 1787828582] req@ffff9e680a2d4780 x1874668790514304/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828582.594059:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790514688/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828582.594066:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790514304/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828582.594078:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828582.594082:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828582.594083:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828582.594088:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828582.594089:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828582.594096:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790514304:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828582.594096:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5219 00000100:00080000:3.0:1787828582.594102:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828582.594109:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828582.594113:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828582.594119:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790514432:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828582.594126:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790514816/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828582.594139:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828582.594152:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828582/real 1787828582] req@ffff9e680a2d4000 x1874668790514432/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828582.594152:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828582.594161:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828582.594165:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828582.594172:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790514432/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828582.594173:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5219 00000100:00080000:3.0:1787828582.594181:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828582.594185:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828582.594189:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787828582.594191:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828582.594197:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790514432:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787828582.594198:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828582.594203:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790514560:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828582.594216:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790514944/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828582.594222:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828582.594233:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828582.594234:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828582/real 1787828582] req@ffff9e680a2d6580 x1874668790514560/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828582.594243:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828582.594247:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828582.594252:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790514560/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828582.594255:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5219 00000100:00080000:3.0:1787828582.594261:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828582.594263:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828582.594268:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787828582.594269:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828582.594275:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790514560:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787828582.594278:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828582.594282:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790514688:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828582.594295:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790515072/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828582.594300:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828582.594310:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828582/real 1787828582] req@ffff9e680a2d43c0 x1874668790514688/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828582.594311:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828582.594320:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828582.594324:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828582.594329:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790514688/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828582.594333:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5219 00000100:00080000:3.0:1787828582.594341:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828582.594344:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828582.594349:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828582.594354:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787828582.594359:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828582.594365:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790514688:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828582.594370:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790515200/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828582.594383:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828582.594388:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787828582.594389:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790514816:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787828582.594391:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828582.594399:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5219 00000100:00080000:3.0:1787828582.594404:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828582.594407:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828582.594412:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828582.594421:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828582.594426:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790515328/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828582.594437:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828582/real 1787828582] req@ffff9e680a2d5a40 x1874668790514816/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828582.594453:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790514816/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828582.594460:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828582.594463:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828582.594466:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790514816:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828582.594470:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790514944:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828582.594485:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828582.594492:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828582/real 1787828582] req@ffff9e680a2d70c0 x1874668790514944/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828582.594500:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790514944/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828582.594506:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828582.594508:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828582.594510:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790514944:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828582.594513:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790515072:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828582.594523:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828582.594529:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828582/real 1787828582] req@ffff9e680a2d4780 x1874668790515072/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828582.594536:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790515072/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828582.594541:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828582.594542:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828582.594544:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790515072:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828582.594547:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790515200:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828582.594557:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828582.594582:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828582/real 1787828582] req@ffff9e680a2d7c00 x1874668790515200/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828582.594590:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790515200/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828582.594600:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828582.594602:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828582.594604:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790515200:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828582.594615:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790515328:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828582.594628:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828582.594635:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828582/real 1787828582] req@ffff9e680a2d6d00 x1874668790515328/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828582.594642:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790515328/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828582.594647:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828582.594648:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828582.594650:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790515328:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828583.105869:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828583.297887:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828584.129829:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828584.321911:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828585.153780:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828585.345869:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828586.177831:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828586.370021:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828587.201778:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828587.393906:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828587.713757:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828587.713765:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828587.713769:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828587.713777:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5224 00000100:00080000:3.0:1787828587.713786:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828587.713792:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828587.713823:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790515456/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828587.713847:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828587.713850:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828587.713852:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828587.713856:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5224 00000100:00080000:3.0:1787828587.713859:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828587.713862:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828587.713870:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828587.713883:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790515456:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828587.713884:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790515584/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828587.713897:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828587.713900:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828587.713901:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828587.713904:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5224 00000100:00080000:3.0:1787828587.713906:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828587.713909:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828587.713912:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828587.713922:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790515712/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828587.713929:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828587.713932:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828587.713933:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828587.713936:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5224 00000100:00080000:3.0:1787828587.713939:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828587.713942:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828587.713945:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828587.713954:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828587.713955:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790515840/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828587.713964:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828587.713966:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828587.713969:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828587.713975:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5224 00000100:00080000:3.0:1787828587.713977:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828587.713981:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828587.713989:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828587.714006:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828587/real 1787828587] req@ffff9e680a2d52c0 x1874668790515456/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828587.714007:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790515968/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828587.714033:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828587.714036:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828587.714038:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828587.714044:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790515456/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828587.714044:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5224 00000100:00080000:0.0:1787828587.714054:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828587.714058:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828587.714061:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790515456:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787828587.714082:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790515584:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828587.714100:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828587.714112:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828587.714114:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828587/real 1787828587] req@ffff9e680a2d4f00 x1874668790515584/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828587.714119:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828587.714128:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828587.714131:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790515584/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828587.714139:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828587.714141:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790516096/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828587.714144:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828587.714151:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790515584:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828587.714161:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790515712:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828587.714170:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828587.714173:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828587.714175:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828587.714178:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5224 00000800:00000100:0.0:1787828587.714182:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828587.714184:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828587.714189:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828587.714193:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828587.714194:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828587/real 1787828587] req@ffff9e680a2d5e00 x1874668790515712/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828587.714204:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790515712/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828587.714209:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828587.714211:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828587.714213:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790515712:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828587.714217:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790515840:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828587.714228:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828587.714235:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828587/real 1787828587] req@ffff9e680a2d7480 x1874668790515840/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828587.714242:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790515840/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828587.714248:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828587.714249:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828587.714251:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790515840:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828587.714254:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790515968:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828587.714265:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828587.714273:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828587/real 1787828587] req@ffff9e680a2d4b40 x1874668790515968/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828587.714281:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790515968/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828587.714286:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828587.714288:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828587.714290:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790515968:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828587.714303:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790516096:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828587.714310:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790516224/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828587.714318:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828587.714323:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828587.714327:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828587.714329:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828587.714345:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828587/real 1787828587] req@ffff9e680a2d61c0 x1874668790516096/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828587.714361:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790516096/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828587.714369:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5224 00000100:00080000:3.0:1787828587.714376:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828587.714384:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828587.714391:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787828587.714394:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828587.714403:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828587.714408:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790516352/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828587.714409:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790516096:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828587.714422:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828587.714425:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787828587.714428:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790516224:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828587.714429:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828587.714437:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5224 00000100:00080000:3.0:1787828587.714450:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828587.714452:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828587.714456:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828587.714467:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828587.714468:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790516480/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828587.714479:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828587/real 1787828587] req@ffff9e680a2d52c0 x1874668790516224/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828587.714489:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790516224/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828587.714495:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828587.714498:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828587.714500:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790516224:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828587.714505:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790516352:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828587.714515:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828587.714521:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828587/real 1787828587] req@ffff9e680a2d6d00 x1874668790516352/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828587.714548:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790516352/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828587.714554:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828587.714556:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828587.714559:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790516352:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828587.714568:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790516480:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828587.714583:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828587.714593:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828587/real 1787828587] req@ffff9e680a2d7c00 x1874668790516480/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828587.714600:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790516480/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828587.714605:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828587.714607:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828587.714609:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790516480:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828588.225762:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828588.417850:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828589.249751:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828589.441877:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828590.273786:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828590.465910:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828591.297750:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828591.489800:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828592.321788:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828592.513826:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828592.833772:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828592.833780:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828592.833784:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828592.833793:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5229 00000100:00080000:3.0:1787828592.833803:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828592.833811:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828592.833849:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790516608/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828592.833871:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828592.833874:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828592.833875:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828592.833879:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5229 00000100:00080000:3.0:1787828592.833882:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828592.833886:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828592.833896:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828592.833900:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790516608:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828592.833909:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790516736/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828592.833919:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828592.833922:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828592.833923:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828592.833926:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5229 00000100:00080000:3.0:1787828592.833929:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828592.833931:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828592.833934:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828592.833942:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790516864/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828592.833948:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828592.833951:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828592.833952:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828592.833954:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5229 00000100:00080000:3.0:1787828592.833957:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828592.833961:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828592.833964:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828592.833969:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828592.833978:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790516992/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828592.833984:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828592.833987:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828592.833988:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828592.833990:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5229 00000100:00080000:3.0:1787828592.833993:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828592.833995:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828592.833999:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828592.834008:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828592/real 1787828592] req@ffff9e680a2d4780 x1874668790516608/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828592.834009:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790517120/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828592.834040:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828592.834043:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828592.834044:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828592.834048:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790516608/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828592.834049:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5229 00000100:00080000:3.0:1787828592.834054:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828592.834061:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828592.834067:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787828592.834069:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828592.834074:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828592.834077:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790517248/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828592.834081:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790516608:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828592.834093:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828592.834097:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828592.834099:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828592.834104:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5229 00000100:00100000:0.0:1787828592.834108:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790516736:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828592.834109:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828592.834118:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828592.834125:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828592.834131:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828592.834143:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790517376/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828592.834144:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828592/real 1787828592] req@ffff9e680a2d70c0 x1874668790516736/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828592.834172:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790516736/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828592.834178:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828592.834183:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828592.834184:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828592.834187:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828592.834191:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828592.834195:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5229 00000100:00100000:0.0:1787828592.834199:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790516736:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828592.834201:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828592.834209:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828592.834211:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790516864:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787828592.834214:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828592.834231:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828592.834237:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790517504/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828592.834242:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828592/real 1787828592] req@ffff9e680a2d5a40 x1874668790516864/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828592.834253:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828592.834257:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787828592.834260:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790516864/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828592.834260:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828592.834268:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5229 00000100:00080000:3.0:1787828592.834274:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828592.834278:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828592.834280:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787828592.834284:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:3.0:1787828592.834285:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828592.834289:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790516864:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828592.834295:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790516992:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828592.834300:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790517632/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828592.834315:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828592.834323:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828592/real 1787828592] req@ffff9e680a2d43c0 x1874668790516992/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828592.834331:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790516992/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828592.834337:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828592.834339:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828592.834341:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790516992:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828592.834344:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790517120:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828592.834354:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828592.834359:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828592/real 1787828592] req@ffff9e680a2d6580 x1874668790517120/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828592.834366:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790517120/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828592.834371:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828592.834373:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828592.834375:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790517120:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828592.834378:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790517248:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828592.834387:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828592.834392:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828592/real 1787828592] req@ffff9e680a2d4000 x1874668790517248/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828592.834415:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790517248/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828592.834421:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828592.834423:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828592.834425:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790517248:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828592.834440:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790517376:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828592.834452:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828592.834458:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828592/real 1787828592] req@ffff9e680a2d4780 x1874668790517376/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828592.834466:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790517376/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828592.834471:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828592.834472:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828592.834474:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790517376:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828592.834478:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790517504:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828592.834500:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828592.834524:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828592/real 1787828592] req@ffff9e680a2d7c00 x1874668790517504/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828592.834532:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790517504/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828592.834537:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828592.834545:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828592.834548:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790517504:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828592.834552:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790517632:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828592.834565:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828592.834572:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828592/real 1787828592] req@ffff9e680a2d6d00 x1874668790517632/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828592.834579:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790517632/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828592.834584:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828592.834586:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828592.834588:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790517632:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828593.345803:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828593.537831:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828594.369770:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828594.561847:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828595.393808:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828595.585846:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828596.417802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828596.609840:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828597.441792:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828597.633827:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828597.953821:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828597.953830:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828597.953849:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828597.953858:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5234 00000100:00080000:0.0:1787828597.953867:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828597.953873:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828597.953907:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790517760/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828597.953930:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828597.953933:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828597.953934:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828597.953937:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5234 00000100:00080000:0.0:1787828597.953939:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828597.953941:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828597.953949:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828597.953958:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790517888/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828597.953966:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828597.953969:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828597.953970:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828597.953973:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5234 00000100:00080000:0.0:1787828597.953974:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828597.953976:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828597.953979:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828597.953985:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790518016/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828597.953993:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790517760:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828597.953995:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828597.953999:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828597.954001:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828597.954007:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5234 00000100:00080000:0.0:1787828597.954010:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828597.954013:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828597.954015:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828597.954041:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790518144/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828597.954047:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828597.954049:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828597.954050:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828597.954052:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5234 00000100:00080000:0.0:1787828597.954056:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828597.954058:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828597.954062:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828597.954070:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790518272/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828597.954075:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828597.954077:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828597.954079:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828597.954084:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828597.954086:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5234 00000100:00080000:0.0:1787828597.954090:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828597.954093:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828597.954097:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828597.954105:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790518400/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828597.954110:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828597.954113:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828597.954114:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828597.954116:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5234 00000100:00080000:0.0:1787828597.954119:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828597.954122:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828597.954124:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828597.954131:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828597/real 1787828597] req@ffff9e6803e9b840 x1874668790517760/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828597.954133:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790518528/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828597.954165:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828597.954167:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828597.954168:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828597.954173:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790517760/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828597.954174:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5234 00000100:00080000:0.0:1787828597.954178:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828597.954184:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828597.954190:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828597.954196:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828597.954200:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790518656/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828597.954205:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787828597.954213:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828597.954215:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790517760:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828597.954216:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828597.954220:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828597.954227:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5234 00000100:00080000:0.0:1787828597.954230:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828597.954233:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790517888:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828597.954234:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828597.954258:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828597.954267:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790518784/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828597.954293:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828597.954308:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828597/real 1787828597] req@ffff9e6803e9bc00 x1874668790517888/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828597.954321:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790517888/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828597.954331:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828597.954334:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828597.954338:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790517888:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828597.954361:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790518016:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828597.954380:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828597.954391:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828597/real 1787828597] req@ffff9e6803e98780 x1874668790518016/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828597.954404:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790518016/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828597.954412:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828597.954416:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828597.954420:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790518016:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828597.954426:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790518144:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828597.954442:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828597.954469:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828597/real 1787828597] req@ffff9e6803e98b40 x1874668790518144/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828597.954483:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790518144/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828597.954493:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828597.954496:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828597.954499:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790518144:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828597.954506:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790518272:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828597.954524:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828597.954535:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828597/real 1787828597] req@ffff9e6803e9a1c0 x1874668790518272/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828597.954547:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790518272/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828597.954556:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828597.954559:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828597.954563:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790518272:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828597.954570:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790518400:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828597.954585:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828597.954595:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828597/real 1787828597] req@ffff9e6803e9b480 x1874668790518400/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828597.954607:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790518400/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828597.954617:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828597.954621:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828597.954625:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790518400:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828597.954630:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790518528:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828597.954647:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828597.954656:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828597/real 1787828597] req@ffff9e6803e9a580 x1874668790518528/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828597.954668:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790518528/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828597.954677:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828597.954681:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828597.954685:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790518528:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828597.954691:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790518656:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828597.954761:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828597.954772:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828597/real 1787828597] req@ffff9e6803e99680 x1874668790518656/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828597.954784:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790518656/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828597.954793:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828597.954812:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828597.954816:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790518656:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828597.954822:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790518784:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828597.954838:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828597.954848:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828597/real 1787828597] req@ffff9e6803e992c0 x1874668790518784/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828597.954861:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790518784/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828622 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828597.954869:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828597.954872:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828597.954876:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790518784:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828598.465806:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828598.657862:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828599.489809:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828599.681880:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828600.513758:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828600.705851:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828601.537751:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828601.729879:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828602.561785:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828602.753814:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828603.073815:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828603.073826:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828603.073832:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828603.073842:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5239 00000100:00080000:2.0:1787828603.073854:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828603.073862:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828603.073899:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790518912/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828603.073927:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828603.073932:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828603.073934:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828603.073939:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5239 00000100:00080000:2.0:1787828603.073944:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828603.073949:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828603.073959:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828603.073967:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790518912:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787828603.073975:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790519040/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828603.073988:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828603.073992:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828603.073994:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828603.073998:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5239 00000100:00080000:2.0:1787828603.074002:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828603.074006:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828603.074012:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828603.074025:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828603.074029:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790519168/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828603.074048:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828603.074052:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828603.074054:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828603.074059:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5239 00000100:00100000:0.0:1787828603.074061:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828603/real 1787828603] req@ffff9e68044221c0 x1874668790518912/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828603.074065:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828603.074074:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828603.074084:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828603.074092:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790518912/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828603.074101:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790519296/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828603.074102:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828603.074109:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828603.074115:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790518912:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828603.074124:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828603.074128:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828603.074131:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828603.074135:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5239 00000100:00080000:2.0:1787828603.074139:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828603.074143:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828603.074146:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790519040:192.168.203.109@tcp:38:kworker.0 00000008:00100000:2.0:1787828603.074151:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828603.074168:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790519424/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828603.074169:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828603.074181:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828603/real 1787828603] req@ffff9e68044230c0 x1874668790519040/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828603.074184:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828603.074194:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828603.074198:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828603.074201:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790519040/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828603.074205:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5239 00000100:00080000:2.0:1787828603.074214:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828603.074215:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828603.074223:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787828603.074224:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828603.074230:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790519040:192.168.203.109@tcp:38:kworker.0 00000008:00100000:2.0:1787828603.074234:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828603.074236:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790519168:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828603.074254:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790519552/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828603.074257:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828603.074267:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828603/real 1787828603] req@ffff9e68044212c0 x1874668790519168/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828603.074272:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828603.074279:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787828603.074280:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790519168/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828603.074282:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828603.074293:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5239 00000100:00080000:2.0:1787828603.074301:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828603.074303:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828603.074308:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787828603.074311:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828603.074313:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790519168:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828603.074321:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790519296:192.168.203.109@tcp:8:kworker.0 00000008:00100000:2.0:1787828603.074321:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828603.074342:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790519680/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828603.074343:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828603.074352:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828603/real 1787828603] req@ffff9e6804423480 x1874668790519296/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828603.074358:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828603.074364:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790519296/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828603.074364:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828603.074369:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828603.074377:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828603.074379:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5239 00000100:00080000:0.0:1787828603.074382:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828603.074387:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790519296:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787828603.074388:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828603.074396:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828603.074401:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790519424:192.168.203.109@tcp:8:kworker.0 00000008:00100000:2.0:1787828603.074402:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828603.074434:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828603.074434:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790519808/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828603.074447:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828603/real 1787828603] req@ffff9e68044221c0 x1874668790519424/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828603.074466:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828603.074472:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790519424/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828603.074473:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828603.074478:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828603.074484:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828603.074488:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5239 00000100:00080000:0.0:1787828603.074490:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787828603.074496:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828603.074497:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790519424:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787828603.074506:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828603.074507:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790519552:192.168.203.109@tcp:38:kworker.0 00000008:00100000:2.0:1787828603.074516:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828603.074528:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828603.074534:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790519936/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828603.074538:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828603/real 1787828603] req@ffff9e6804423840 x1874668790519552/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828603.074565:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790519552/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828603.074572:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828603.074574:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828603.074577:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790519552:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828603.074582:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790519680:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828603.074592:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828603.074598:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828603/real 1787828603] req@ffff9e6804420b40 x1874668790519680/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828603.074605:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790519680/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828603.074610:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828603.074612:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828603.074614:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790519680:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828603.074623:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790519808:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828603.074632:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828603.074639:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828603/real 1787828603] req@ffff9e6804422580 x1874668790519808/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828603.074646:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790519808/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828603.074651:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828603.074653:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828603.074655:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790519808:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828603.074658:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790519936:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828603.074667:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828603.074672:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828603/real 1787828603] req@ffff9e6804421680 x1874668790519936/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828603.074683:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790519936/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828603.074691:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828603.074715:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828603.074719:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790519936:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828603.585767:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828603.777848:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828604.609761:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828604.801879:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828605.633748:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828605.825869:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828606.657768:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828606.849859:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828607.681806:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828607.873876:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828608.193757:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828608.193765:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828608.193769:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828608.193777:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5244 00000100:00080000:2.0:1787828608.193785:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828608.193791:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828608.193820:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790520064/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828608.193839:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828608.193842:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828608.193843:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828608.193847:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5244 00000100:00080000:2.0:1787828608.193849:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828608.193852:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828608.193862:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828608.193876:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790520192/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828608.193884:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790520064:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828608.193895:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828608.193900:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828608.193903:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828608.193908:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5244 00000100:00080000:2.0:1787828608.193911:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828608.193914:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828608.193919:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828608.193929:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790520320/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828608.193937:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828608.193940:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828608.193941:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828608.193944:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5244 00000100:00080000:2.0:1787828608.193947:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828608.193952:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828608.193953:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828608.193959:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828608.193968:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790520448/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828608.193974:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828608.193977:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828608.193978:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828608.193981:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5244 00000100:00080000:2.0:1787828608.193984:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828608.193987:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828608.193991:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828608.193994:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828608/real 1787828608] req@ffff9e6804420780 x1874668790520064/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828608.194002:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790520576/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828608.194013:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828608.194016:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790520064/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828608.194020:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828608.194026:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828608.194033:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828608.194035:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5244 00000100:00080000:0.0:1787828608.194042:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787828608.194042:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828608.194049:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790520064:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828608.194053:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828608.194062:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828608.194079:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790520192:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828608.194086:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790520704/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828608.194106:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828608.194109:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828608.194110:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828608.194116:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5244 00000800:00000100:0.0:1787828608.194117:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828608.194120:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828608.194125:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828608.194129:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828608.194139:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828608/real 1787828608] req@ffff9e6804420f00 x1874668790520192/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828608.194140:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790520832/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828608.194173:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828608.194175:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828608.194177:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828608.194180:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790520192/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828608.194184:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5244 00000100:00080000:2.0:1787828608.194190:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828608.194197:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787828608.194198:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787828608.194200:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787828608.194204:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828608.194208:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790520192:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828608.194212:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790520960/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828608.194216:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790520320:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828608.194228:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828608.194230:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828608.194232:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828608.194236:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5244 00000100:00080000:2.0:1787828608.194240:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828608.194244:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828608.194245:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828608.194251:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828608.194259:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828608/real 1787828608] req@ffff9e6804421e00 x1874668790520320/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828608.194280:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790521088/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828608.194284:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790520320/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828608.194312:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828608.194316:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828608.194319:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790520320:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828608.194325:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790520448:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828608.194349:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828608.194355:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828608/real 1787828608] req@ffff9e6804422940 x1874668790520448/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828608.194363:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790520448/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828608.194368:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828608.194370:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828608.194372:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790520448:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828608.194375:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790520576:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828608.194402:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828608.194410:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828608/real 1787828608] req@ffff9e6804422d00 x1874668790520576/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828608.194418:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790520576/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828608.194424:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828608.194436:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828608.194438:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790520576:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828608.194450:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790520704:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828608.194461:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828608.194467:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828608/real 1787828608] req@ffff9e6804420780 x1874668790520704/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828608.194475:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790520704/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828608.194481:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828608.194483:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828608.194485:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790520704:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828608.194488:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790520832:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828608.194498:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828608.194504:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828608/real 1787828608] req@ffff9e6804421680 x1874668790520832/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828608.194511:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790520832/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828608.194515:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828608.194517:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828608.194519:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790520832:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828608.194522:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790520960:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828608.194531:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828608.194536:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828608/real 1787828608] req@ffff9e6804422580 x1874668790520960/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828608.194543:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790520960/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828608.194552:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828608.194554:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828608.194556:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790520960:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828608.194559:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790521088:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828608.194568:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828608.194573:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828608/real 1787828608] req@ffff9e6804420b40 x1874668790521088/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828608.194580:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790521088/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828608.194584:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828608.194586:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828608.194588:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790521088:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828608.705756:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828608.897869:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828609.729796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828609.921841:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828610.753752:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828610.945872:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828611.777801:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828611.969841:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828612.801800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828612.993905:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828613.314772:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828613.314782:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828613.314786:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828613.314796:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5250 00000100:00080000:3.0:1787828613.314806:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828613.314859:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828613.314894:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790521216/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828613.314927:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828613.314934:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828613.314937:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828613.314942:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5250 00000100:00080000:3.0:1787828613.314948:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828613.314954:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828613.314963:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790521216:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787828613.314964:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828613.314983:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790521344/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828613.314998:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828613.315002:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828613.315005:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828613.315010:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5250 00000100:00080000:3.0:1787828613.315015:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828613.315019:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828613.315025:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828613.315032:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828613.315041:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790521472/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828613.315052:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828613.315056:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828613.315058:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828613.315062:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5250 00000100:00080000:3.0:1787828613.315065:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828613.315070:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828613.315076:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828613.315088:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828613/real 1787828613] req@ffff9e680a2d52c0 x1874668790521216/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828613.315092:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790521600/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828613.315119:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828613.315123:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828613.315126:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828613.315128:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790521216/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828613.315133:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5250 00000100:00080000:3.0:1787828613.315138:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828613.315142:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828613.315147:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787828613.315159:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828613.315167:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790521728/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828613.315168:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828613.315176:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790521216:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828613.315190:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828613.315195:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828613.315197:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828613.315202:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5250 00000100:00080000:3.0:1787828613.315206:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828613.315212:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790521344:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828613.315213:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828613.315223:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828613.315240:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790521856/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828613.315251:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828613.315253:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828613.315258:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828613.315261:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828613.315266:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5250 00000100:00100000:0.0:1787828613.315272:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828613/real 1787828613] req@ffff9e680a2d61c0 x1874668790521344/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828613.315273:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828613.315282:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828613.315291:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828613.315299:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790521344/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828613.315307:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790521984/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828613.315325:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828613.315332:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828613.315334:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828613.315334:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828613.315359:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828613.315360:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5250 00000100:00080000:3.0:1787828613.315368:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828613.315370:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790521344:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828613.315377:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828613.315384:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790521472:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787828613.315386:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828613.315404:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790522112/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828613.315415:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828613.315418:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828613.315423:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828613.315425:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828613.315430:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5250 00000100:00080000:3.0:1787828613.315435:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828613.315439:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828613.315444:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828613/real 1787828613] req@ffff9e680a2d4b40 x1874668790521472/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828613.315447:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828613.315467:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790522240/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828613.315469:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790521472/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828613.315509:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828613.315515:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828613.315519:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790521472:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828613.315527:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790521600:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828613.315550:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828613.315562:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828613/real 1787828613] req@ffff9e680a2d7480 x1874668790521600/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828613.315576:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790521600/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828613.315586:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828613.315589:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828613.315592:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790521600:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828613.315598:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790521728:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828613.315618:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828613.315630:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828613/real 1787828613] req@ffff9e680a2d5e00 x1874668790521728/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828613.315643:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790521728/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828613.315651:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828613.315654:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828613.315657:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790521728:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828613.315678:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790521856:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828613.315733:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828613.315747:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828613/real 1787828613] req@ffff9e680a2d4f00 x1874668790521856/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828613.315760:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790521856/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828613.315777:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828613.315781:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828613.315784:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790521856:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828613.315790:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790521984:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828613.315811:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828613.315824:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828613/real 1787828613] req@ffff9e680a2d52c0 x1874668790521984/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828613.315836:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790521984/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828613.315845:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828613.315848:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828613.315851:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790521984:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828613.315867:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790522112:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828613.315887:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828613.315898:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828613/real 1787828613] req@ffff9e680a2d6d00 x1874668790522112/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828613.315910:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790522112/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828613.315918:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828613.315921:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828613.315925:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790522112:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828613.315931:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790522240:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828613.315950:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828613.315962:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828613/real 1787828613] req@ffff9e680a2d7c00 x1874668790522240/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828613.315974:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790522240/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828613.315982:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828613.315985:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828613.315989:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790522240:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828613.825803:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828614.018834:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828614.849769:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828615.041852:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828615.873798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828616.065894:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828616.897780:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828617.089858:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828617.921794:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828618.113840:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828618.434990:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828618.434999:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828618.435003:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828618.435011:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5255 00000100:00080000:1.0:1787828618.435021:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828618.435027:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828618.435091:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790522368/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828618.435114:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828618.435117:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828618.435119:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828618.435121:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5255 00000100:00080000:1.0:1787828618.435125:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828618.435128:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828618.435137:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828618.435148:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790522496/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828618.435156:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828618.435158:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828618.435160:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828618.435163:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5255 00000100:00080000:1.0:1787828618.435165:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828618.435169:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828618.435172:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828618.435179:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790522624/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828618.435187:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828618.435189:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828618.435192:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828618.435201:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5255 00000100:00100000:0.0:1787828618.435202:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790522368:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828618.435204:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828618.435209:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828618.435219:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828618.435227:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790522752/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828618.435234:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828618.435236:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828618.435237:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828618.435239:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5255 00000100:00080000:1.0:1787828618.435243:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828618.435245:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828618.435248:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828618.435255:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790522880/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828618.435260:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828618.435264:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828618.435265:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828618.435268:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5255 00000100:00080000:1.0:1787828618.435293:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828618.435296:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828618.435300:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828618.435323:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828618.435336:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790523008/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828618.435344:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828618.435346:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828618.435348:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828618.435350:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5255 00000100:00080000:1.0:1787828618.435353:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828618.435355:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828618.435358:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828618.435364:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790523136/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828618.435369:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828618.435371:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828618.435372:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828618.435374:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5255 00000100:00080000:1.0:1787828618.435376:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828618.435378:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828618.435381:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828618.435386:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790523264/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828618.435391:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828618.435393:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828618.435394:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828618.435397:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5255 00000100:00080000:1.0:1787828618.435398:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828618.435400:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828618.435403:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828618.435408:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790523392/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828618.436584:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828618/real 1787828618] req@ffff9e6831aaad00 x1874668790522368/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828618.436606:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790522368/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828618.436617:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828618.436621:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828618.436625:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790522368:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787828618.436636:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790522496:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828618.436683:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828618.436712:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828618/real 1787828618] req@ffff9e6831aabc00 x1874668790522496/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828618.436725:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790522496/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828618.436736:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828618.436739:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828618.436743:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790522496:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828618.436766:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790522624:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828618.436786:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828618.436797:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828618/real 1787828618] req@ffff9e6831aab480 x1874668790522624/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828618.436809:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790522624/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828618.436818:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828618.436821:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828618.436825:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790522624:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828618.436831:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790522752:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828618.436848:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828618.436857:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828618/real 1787828618] req@ffff9e6831aa9a40 x1874668790522752/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828618.436868:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790522752/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828618.436878:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828618.436882:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828618.436885:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790522752:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828618.436891:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790522880:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828618.436907:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828618.436916:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828618/real 1787828618] req@ffff9e6831aa8000 x1874668790522880/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828618.436928:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790522880/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828618.436936:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828618.436939:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828618.436943:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790522880:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828618.436949:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790523008:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828618.436968:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828618.436978:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828618/real 1787828618] req@ffff9e6831aaa1c0 x1874668790523008/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828618.436990:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790523008/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828618.437000:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828618.437003:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828618.437007:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790523008:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828618.437013:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790523136:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828618.437029:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828618.437039:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828618/real 1787828618] req@ffff9e6831aa8780 x1874668790523136/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828618.437061:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790523136/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828618.437070:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828618.437073:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828618.437077:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790523136:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828618.437084:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790523264:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828618.437101:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828618.437110:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828618/real 1787828618] req@ffff9e6831aaa940 x1874668790523264/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828618.437122:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790523264/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828618.437130:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828618.437133:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828618.437137:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790523264:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828618.437143:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790523392:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828618.437158:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828618.437167:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828618/real 1787828618] req@ffff9e6831aa8b40 x1874668790523392/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828618.437178:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790523392/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828618.437187:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828618.437190:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828618.437193:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790523392:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828618.945792:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828619.137864:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828619.969770:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828620.161842:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828620.993777:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828621.185883:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828622.017807:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828622.209854:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828623.041775:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828623.233850:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828623.553771:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828623.553783:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828623.553787:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828623.553795:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5260 00000100:00080000:1.0:1787828623.553805:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828623.553811:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828623.553843:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790523520/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828623.553869:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828623.553872:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828623.553873:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828623.553876:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5260 00000100:00080000:1.0:1787828623.553879:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828623.553881:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828623.553888:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828623.553898:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790523648/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828623.553905:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828623.553908:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828623.553909:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828623.553911:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5260 00000100:00080000:1.0:1787828623.553913:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828623.553924:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828623.553932:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828623.553943:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790523520:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828623.553947:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790523776/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828623.553960:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828623.553963:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828623.553966:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828623.553971:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5260 00000100:00080000:1.0:1787828623.553975:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828623.553979:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828623.553985:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828623.553996:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790523904/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828623.554003:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828623.554005:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828623.554007:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828623.554009:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5260 00000100:00080000:1.0:1787828623.554012:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828623.554014:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828623.554018:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828623.554027:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828623.554030:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790524032/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828623.554038:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828623.554041:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828623.554042:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828623.554045:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5260 00000100:00080000:1.0:1787828623.554048:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828623.554051:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828623.554053:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828623.554069:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790524160/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828623.554073:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828623/real 1787828623] req@ffff9e6831aa9e00 x1874668790523520/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828623.554087:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828623.554095:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828623.554098:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828623.554103:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5260 00000100:00080000:1.0:1787828623.554106:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828623.554112:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828623.554114:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790523520/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828623.554116:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828623.554129:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790524288/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828623.554133:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828623.554137:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828623.554142:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828623.554143:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828623.554144:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828623.554152:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790523520:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828623.554154:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5260 00000100:00080000:1.0:1787828623.554162:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828623.554167:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790523648:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828623.554170:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828623.554179:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828623.554194:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790524416/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828623.554200:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828623.554208:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828623.554212:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828623.554213:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828623.554217:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828623/real 1787828623] req@ffff9e6831aa9680 x1874668790523648/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828623.554219:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5260 00000100:00080000:1.0:1787828623.554226:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828623.554232:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828623.554236:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790523648/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828623.554238:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828623.554249:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828623.554253:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790524544/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828623.554254:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828623.554262:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790523648:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828623.554305:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790523776:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828623.554322:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828623.554331:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828623/real 1787828623] req@ffff9e6831aaa580 x1874668790523776/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828623.554339:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790523776/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828623.554349:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828623.554351:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828623.554353:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790523776:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828623.554357:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790523904:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828623.554367:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828623.554373:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828623/real 1787828623] req@ffff9e6831aa8f00 x1874668790523904/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828623.554380:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790523904/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828623.554386:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828623.554387:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828623.554389:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790523904:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828623.554393:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790524032:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828623.554402:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828623.554407:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828623/real 1787828623] req@ffff9e6831aa8b40 x1874668790524032/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828623.554414:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790524032/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828623.554419:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828623.554421:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828623.554424:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790524032:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828623.554427:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790524160:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828623.554437:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828623.554442:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828623/real 1787828623] req@ffff9e6831aaa940 x1874668790524160/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828623.554449:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790524160/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828623.554456:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828623.554457:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828623.554459:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790524160:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828623.554462:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790524288:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828623.554472:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828623.554478:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828623/real 1787828623] req@ffff9e6831aa8780 x1874668790524288/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828623.554485:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790524288/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828623.554489:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828623.554491:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828623.554493:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790524288:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828623.554496:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790524416:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828623.554505:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828623.554511:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828623/real 1787828623] req@ffff9e6831aaa1c0 x1874668790524416/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828623.554519:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790524416/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828623.554524:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828623.554525:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828623.554527:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790524416:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828623.554530:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790524544:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828623.554540:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828623.554545:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828623/real 1787828623] req@ffff9e6831aa8000 x1874668790524544/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828623.554552:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790524544/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828623.554556:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828623.554558:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828623.554560:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790524544:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828624.065773:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828624.257816:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828625.089766:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828625.281886:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828626.113796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828626.305864:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828627.137759:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828627.329878:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828628.161788:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828628.353875:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828628.673776:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828628.673785:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828628.673789:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828628.673798:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5265 00000100:00080000:1.0:1787828628.673808:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828628.673814:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828628.673848:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790524672/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828628.673869:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828628.673872:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828628.673874:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828628.673876:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5265 00000100:00080000:1.0:1787828628.673879:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828628.673882:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828628.673890:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828628.673902:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790524800/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828628.673902:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790524672:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828628.673916:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828628.673918:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828628.673920:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828628.673923:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5265 00000100:00080000:1.0:1787828628.673925:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828628.673927:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828628.673930:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828628.673940:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790524928/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828628.673946:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828628.673948:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828628.673949:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828628.673952:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5265 00000100:00080000:1.0:1787828628.673964:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828628.673967:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828628.673969:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828628.673979:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828628.673980:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790525056/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828628.673993:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828628.673997:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828628.673999:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828628.674003:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5265 00000100:00080000:1.0:1787828628.674008:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828628.674013:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828628.674019:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828628.674023:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828628/real 1787828628] req@ffff9e6831aa9a40 x1874668790524672/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828628.674035:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790525184/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828628.674046:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828628.674049:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790524672/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828628.674050:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828628.674054:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828628.674060:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5265 00000100:00080000:1.0:1787828628.674066:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828628.674068:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828628.674071:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787828628.674076:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:1.0:1787828628.674078:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828628.674080:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790524672:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828628.674089:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790525312/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828628.674096:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828628.674098:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828628.674100:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828628.674104:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790524800:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828628.674106:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5265 00000100:00080000:1.0:1787828628.674111:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828628.674114:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828628.674117:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828628.674127:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790525440/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828628.674134:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828628.674144:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828628.674148:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828628/real 1787828628] req@ffff9e6831aab480 x1874668790524800/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828628.674149:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828628.674152:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828628.674158:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5265 00000100:00080000:1.0:1787828628.674164:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828628.674169:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828628.674172:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828628.674173:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790524800/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828628.674183:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828628.674184:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790525568/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828628.674189:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828628.674197:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790524800:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828628.674200:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828628.674204:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828628.674207:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790524928:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828628.674208:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828628.674215:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5265 00000100:00080000:1.0:1787828628.674218:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828628.674220:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828628.674224:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828628.674231:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828628.674235:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790525696/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828628.674259:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828628/real 1787828628] req@ffff9e6831aabc00 x1874668790524928/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828628.674270:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790524928/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828628.674281:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828628.674283:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828628.674285:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790524928:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828628.674290:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790525056:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828628.674304:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828628.674310:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828628/real 1787828628] req@ffff9e6831aaad00 x1874668790525056/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828628.674317:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790525056/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828628.674323:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828628.674325:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828628.674326:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790525056:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828628.674330:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790525184:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828628.674340:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828628.674345:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828628/real 1787828628] req@ffff9e6831aa8000 x1874668790525184/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828628.674353:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790525184/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828628.674358:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828628.674359:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828628.674362:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790525184:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828628.674365:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790525312:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828628.674375:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828628.674381:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828628/real 1787828628] req@ffff9e6831aaa1c0 x1874668790525312/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828628.674387:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790525312/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828628.674394:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828628.674396:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828628.674397:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790525312:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828628.674409:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790525440:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828628.674420:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828628.674425:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828628/real 1787828628] req@ffff9e6831aa8780 x1874668790525440/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828628.674433:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790525440/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828628.674437:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828628.674439:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828628.674441:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790525440:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828628.674444:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790525568:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828628.674454:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828628.674459:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828628/real 1787828628] req@ffff9e6831aaa940 x1874668790525568/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828628.674466:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790525568/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828628.674471:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828628.674473:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828628.674475:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790525568:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828628.674478:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790525696:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828628.674487:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828628.674492:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828628/real 1787828628] req@ffff9e6831aa8b40 x1874668790525696/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828628.674499:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790525696/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828628.674504:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828628.674506:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828628.674508:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790525696:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828629.185810:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828629.377866:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828630.209795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828630.401840:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828631.233805:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828631.425860:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828632.257811:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828632.449921:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828633.281834:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828633.473906:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828633.793925:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828633.793936:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828633.793941:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828633.793952:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5270 00000100:00080000:3.0:1787828633.793965:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828633.793973:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828633.794018:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790525824/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828633.794062:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828633.794065:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828633.794067:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828633.794070:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5270 00000100:00080000:3.0:1787828633.794073:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828633.794076:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828633.794103:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828633.794115:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790525952/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828633.794124:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828633.794127:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828633.794129:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828633.794135:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5270 00000100:00100000:2.0:1787828633.794139:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790525824:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828633.794139:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828633.794143:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828633.794152:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828633.794162:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790526080/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828633.794170:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828633.794173:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828633.794174:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828633.794177:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5270 00000100:00080000:3.0:1787828633.794179:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828633.794183:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828633.794186:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828633.794194:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790526208/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828633.794201:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828633.794218:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828633.794220:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828633.794224:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5270 00000100:00080000:3.0:1787828633.794230:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828633.794235:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828633.794239:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828633.794248:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790526336/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828633.794255:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828633.794259:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828633.794261:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828633.794264:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5270 00000100:00080000:3.0:1787828633.794268:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828633.794273:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828633.794277:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828633.794303:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790526464/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828633.794311:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828633.794314:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790525952:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828633.794315:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828633.794320:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828633.794331:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5270 00000100:00080000:3.0:1787828633.794334:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828633.794338:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828633.794342:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828633.794355:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790526592/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828633.794364:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828633.794368:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828633.794369:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828633.794373:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5270 00000100:00100000:2.0:1787828633.794377:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790526080:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828633.794377:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828633.794392:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828633.794398:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828633.794414:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790526208:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828633.794420:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790526720/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828633.794430:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828633.794433:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828633.794435:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828633.794440:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5270 00000100:00080000:3.0:1787828633.794444:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828633.794452:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828633.794456:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790526336:192.168.203.109@tcp:8:kworker.0 00000008:00100000:3.0:1787828633.794458:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828633.794473:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790526848/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828633.794511:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790526464:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828633.794542:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790526592:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828633.794566:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790526720:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828633.794588:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790526848:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828634.305846:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828634.497837:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828635.329840:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828635.521864:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828636.353795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828636.545872:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787828636.866024:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:0.0:1787828636.866093:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787828636.866113:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828636.866204:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:0.0:1787828636.866223:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828636.866253:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828636.866262:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828636.866277:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828636.866294:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787828636.866300:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828633/real 1787828636] req@ffff9e680a2d52c0 x1874668790526848/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787828636.866313:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828636.866332:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787828636.866354:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790526848/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828636.866403:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828636.866411:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828636.866418:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790526848:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828636.866430:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828633/real 1787828636] req@ffff9e680a2d4000 x1874668790525952/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828636.866452:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790525952/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828636.866465:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828636.866474:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828636.866479:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790525952:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828636.866489:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828633/real 1787828636] req@ffff9e680a2d6580 x1874668790526080/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828636.866506:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790526080/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828636.866517:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828636.866523:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828636.866527:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790526080:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828636.866537:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828633/real 1787828636] req@ffff9e680a2d43c0 x1874668790526208/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828636.866554:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790526208/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828636.866566:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828636.866570:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828636.866575:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790526208:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828636.866584:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828633/real 1787828636] req@ffff9e680a2d5a40 x1874668790526336/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828636.866599:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790526336/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828636.866611:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828636.866616:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828636.866621:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790526336:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828636.866629:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828633/real 1787828636] req@ffff9e680a2d4780 x1874668790525824/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828636.866646:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790525824/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828636.866658:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828636.866663:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828636.866667:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790525824:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787828636.866748:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828633/real 1787828636] req@ffff9e680a2d70c0 x1874668790526464/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828636.866769:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790526464/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828636.866783:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828636.866788:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828636.866793:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790526464:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828636.866802:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828633/real 1787828636] req@ffff9e680a2d7c00 x1874668790526592/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828636.866817:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790526592/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828636.866828:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828636.866832:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828636.866837:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790526592:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828636.866844:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828633/real 1787828636] req@ffff9e680a2d6d00 x1874668790526720/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828636.866859:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790526720/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828636.866870:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828636.866874:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828636.866878:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790526720:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828637.377871:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828637.569847:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828638.401799:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828638.593836:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828638.913828:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828638.913836:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828638.913839:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828638.913848:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5275 00000100:00080000:0.0:1787828638.913856:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828638.913862:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828638.913892:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790526976/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828638.913918:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828638.913922:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828638.913924:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828638.913928:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5275 00000100:00080000:0.0:1787828638.913932:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828638.913936:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828638.913946:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828638.913975:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790527104/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828638.913987:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828638.913993:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828638.913993:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790526976:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828638.913998:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828638.914006:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5275 00000100:00080000:0.0:1787828638.914015:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828638.914020:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828638.914026:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828638.914040:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790527232/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828638.914049:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828638.914052:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828638.914054:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828638.914058:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5275 00000100:00080000:0.0:1787828638.914063:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828638.914067:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828638.914072:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828638.914078:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828638.914093:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790527360/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828638.914103:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828638.914106:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828638.914109:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828638.914113:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5275 00000100:00080000:0.0:1787828638.914117:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828638.914122:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828638.914127:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828638/real 1787828638] req@ffff9e6803e9ad00 x1874668790526976/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828638.914130:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828638.914150:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790527488/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828638.914158:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790526976/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828638.914183:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828638.914189:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828638.914192:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828638.914193:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828638.914199:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5275 00000100:00080000:2.0:1787828638.914219:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787828638.914222:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828638.914227:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790526976:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828638.914233:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828638.914243:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828638.914257:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790527616/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828638.914267:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790527104:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828638.914286:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828638.914290:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828638.914293:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828638.914298:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5275 00000100:00080000:0.0:1787828638.914303:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828638.914306:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828638.914311:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828638.914318:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828638.914330:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828638/real 1787828638] req@ffff9e6803e98f00 x1874668790527104/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828638.914333:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790527744/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828638.914361:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828638.914364:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790527104/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828638.914366:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828638.914370:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828638.914379:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5275 00000100:00080000:0.0:1787828638.914386:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828638.914390:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828638.914394:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828638.914402:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828638.914409:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790527872/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828638.914409:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828638.914417:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790527104:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828638.914432:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828638.914433:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790527232:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828638.914443:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828638.914447:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828638.914453:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5275 00000100:00080000:0.0:1787828638.914459:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828638.914464:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787828638.914469:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787828638.914471:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828638.914487:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828638/real 1787828638] req@ffff9e6803e99e00 x1874668790527232/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828638.914489:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790528000/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828638.914530:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790527232/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828638.914547:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828638.914555:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828638.914558:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790527232:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828638.914565:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790527360:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828638.914591:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828638.914605:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828638/real 1787828638] req@ffff9e6803e9a940 x1874668790527360/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828638.914618:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790527360/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828638.914627:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828638.914630:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828638.914633:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790527360:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828638.914640:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790527488:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828638.914660:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828638.914671:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828638/real 1787828638] req@ffff9e6803e992c0 x1874668790527488/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828638.914682:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790527488/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828638.914690:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828638.914720:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828638.914725:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790527488:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828638.914745:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790527616:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828638.914769:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828638.914782:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828638/real 1787828638] req@ffff9e6803e99680 x1874668790527616/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828638.914797:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790527616/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828638.914816:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828638.914820:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828638.914824:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790527616:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828638.914830:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790527744:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828638.914849:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828638.914860:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828638/real 1787828638] req@ffff9e6803e9a580 x1874668790527744/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828638.914870:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790527744/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828638.914878:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828638.914882:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828638.914885:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790527744:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828638.914891:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790527872:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828638.914908:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828638.914918:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828638/real 1787828638] req@ffff9e6803e9b480 x1874668790527872/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828638.914928:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790527872/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828638.914935:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828638.914938:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828638.914941:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790527872:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828638.914947:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790528000:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828638.914971:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828638.914982:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828638/real 1787828638] req@ffff9e6803e9a1c0 x1874668790528000/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828638.914994:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790528000/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828663 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828638.915001:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828638.915004:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828638.915007:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790528000:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828639.425805:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828639.617812:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828640.449791:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828640.641907:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828641.473806:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828641.665839:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828642.497757:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828642.689855:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828643.521783:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828643.713841:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828644.033880:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828644.033890:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828644.033895:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828644.033906:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5280 00000100:00080000:3.0:1787828644.033918:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828644.033926:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828644.033963:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790528128/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828644.034014:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828644.034020:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828644.034023:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828644.034032:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5280 00000100:00080000:3.0:1787828644.034039:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828644.034046:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828644.034073:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790528128:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787828644.034078:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828644.034108:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790528256/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828644.034129:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828644.034157:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828644.034175:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828644.034180:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5280 00000100:00080000:3.0:1787828644.034185:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828644.034190:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787828644.034194:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787828644.034198:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828644.034213:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790528384/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828644.034240:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828644.034245:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828644.034249:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828644.034251:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828644/real 1787828644] req@ffff9e680a2d4f00 x1874668790528128/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828644.034258:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5280 00000100:00080000:3.0:1787828644.034270:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828644.034278:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828644.034281:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790528128/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828644.034287:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828644.034296:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828644.034302:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828644.034310:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790528128:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828644.034312:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790528512/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828644.034328:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828644.034333:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828644.034336:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790528256:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828644.034338:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828644.034348:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5280 00000100:00080000:3.0:1787828644.034355:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828644.034361:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787828644.034367:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787828644.034370:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828644.034381:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828644/real 1787828644] req@ffff9e680a2d5e00 x1874668790528256/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828644.034393:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790528640/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828644.034399:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790528256/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828644.034424:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828644.034429:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828644.034430:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828644.034435:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790528256:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828644.034440:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828644.034445:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828644.034446:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790528384:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828644.034453:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5280 00000100:00080000:3.0:1787828644.034461:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828644.034467:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787828644.034472:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787828644.034479:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828644.034486:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828644/real 1787828644] req@ffff9e680a2d7480 x1874668790528384/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828644.034502:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790528384/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828644.034502:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790528768/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828644.034548:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828644.034557:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828644.034558:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828644.034563:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828644.034565:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790528384:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828644.034566:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828644.034591:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5280 00000100:00080000:3.0:1787828644.034595:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828644.034596:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790528512:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787828644.034602:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828644.034608:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828644.034620:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790528896/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828644.034623:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828644.034634:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828644.034638:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828644.034655:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828644.034662:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5280 00000100:00100000:2.0:1787828644.034667:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828644/real 1787828644] req@ffff9e680a2d4b40 x1874668790528512/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828644.034668:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828644.034681:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828644.034689:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790528512/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828644.034690:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828644.034721:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828644.034727:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790529024/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828644.034738:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828644.034742:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828644.034744:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828644.034745:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828644.034755:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790528512:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787828644.034755:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5280 00000100:00080000:3.0:1787828644.034761:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828644.034765:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828644.034773:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828644.034799:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790529152/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828644.034808:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790528640:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828644.034830:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828644.034844:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828644/real 1787828644] req@ffff9e680a2d61c0 x1874668790528640/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828644.034857:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790528640/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828644.034883:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828644.034886:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828644.034892:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790528640:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828644.034899:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790528768:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828644.034919:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828644.034928:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828644/real 1787828644] req@ffff9e680a2d4f00 x1874668790528768/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828644.034936:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790528768/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828644.034943:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828644.034945:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828644.034947:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790528768:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828644.034951:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790528896:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828644.034962:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828644.034968:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828644/real 1787828644] req@ffff9e680a2d6d00 x1874668790528896/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828644.034975:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790528896/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828644.034997:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828644.035000:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828644.035002:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790528896:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828644.035006:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790529024:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828644.035021:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828644.035028:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828644/real 1787828644] req@ffff9e680a2d7c00 x1874668790529024/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828644.035038:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790529024/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828644.035044:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828644.035046:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828644.035049:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790529024:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828644.035062:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790529152:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828644.035076:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828644.035083:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828644/real 1787828644] req@ffff9e680a2d70c0 x1874668790529152/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828644.035092:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790529152/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828644.035099:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828644.035101:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828644.035103:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790529152:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828644.545786:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828644.737862:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828645.569800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828645.761872:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828646.593824:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828646.785971:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828647.617778:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828647.809961:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828648.641798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828648.833856:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828649.153765:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828649.153774:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828649.153778:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828649.153786:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5285 00000100:00080000:3.0:1787828649.153796:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828649.153802:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828649.153838:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790529280/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828649.153867:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828649.153870:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828649.153872:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828649.153875:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5285 00000100:00080000:3.0:1787828649.153878:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828649.153881:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828649.153889:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828649.153899:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790529408/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828649.153908:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828649.153910:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828649.153911:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828649.153913:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5285 00000100:00080000:3.0:1787828649.153916:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828649.153918:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828649.153922:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828649.153929:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790529536/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828649.153937:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828649.153939:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828649.153943:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828649.153944:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790529280:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828649.153948:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5285 00000100:00080000:3.0:1787828649.153953:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828649.153956:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828649.153959:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828649.153969:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790529664/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828649.153975:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828649.153978:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828649.153979:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828649.153982:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5285 00000100:00080000:3.0:1787828649.153994:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828649.153998:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828649.154001:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828649.154008:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828649.154012:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790529792/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828649.154020:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828649.154022:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828649.154024:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828649.154026:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5285 00000100:00080000:3.0:1787828649.154029:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828649.154032:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828649.154035:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828649.154043:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790529920/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828649.154048:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828649/real 1787828649] req@ffff9e680a2d4780 x1874668790529280/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828649.154052:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828649.154060:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828649.154063:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828649.154069:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5285 00000100:00080000:3.0:1787828649.154071:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828649.154075:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790529280/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828649.154076:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828649.154083:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828649.154092:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828649.154106:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790530048/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828649.154114:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828649.154123:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790529280:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828649.154132:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828649.154136:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790529408:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828649.154138:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828649.154141:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828649.154147:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5285 00000100:00080000:3.0:1787828649.154150:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828649.154153:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828649.154158:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828649.154164:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828649.154171:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790530176/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828649.154179:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828649/real 1787828649] req@ffff9e680a2d5a40 x1874668790529408/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828649.154181:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828649.154190:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828649.154193:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828649.154202:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5285 00000100:00100000:2.0:1787828649.154206:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790529408/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828649.154206:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828649.154213:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828649.154220:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828649.154223:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828649.154228:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828649.154231:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790530304/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828649.154235:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790529408:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828649.154264:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790529536:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828649.154278:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828649.154285:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828649/real 1787828649] req@ffff9e680a2d43c0 x1874668790529536/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828649.154292:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790529536/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828649.154297:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828649.154299:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828649.154301:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790529536:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828649.154304:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790529664:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828649.154314:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828649.154320:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828649/real 1787828649] req@ffff9e680a2d6580 x1874668790529664/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828649.154328:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790529664/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828649.154333:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828649.154335:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828649.154336:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790529664:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828649.154340:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790529792:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828649.154356:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828649.154363:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828649/real 1787828649] req@ffff9e680a2d4000 x1874668790529792/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828649.154370:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790529792/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828649.154375:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828649.154377:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828649.154379:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790529792:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828649.154382:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790529920:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828649.154393:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828649.154399:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828649/real 1787828649] req@ffff9e680a2d52c0 x1874668790529920/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828649.154406:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790529920/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828649.154412:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828649.154414:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828649.154416:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790529920:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828649.154419:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790530048:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828649.154429:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828649.154435:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828649/real 1787828649] req@ffff9e680a2d70c0 x1874668790530048/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828649.154442:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790530048/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828649.154447:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828649.154449:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828649.154451:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790530048:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828649.154454:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790530176:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828649.154463:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828649.154469:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828649/real 1787828649] req@ffff9e680a2d7c00 x1874668790530176/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828649.154475:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790530176/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828649.154480:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828649.154482:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828649.154484:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790530176:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828649.154488:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790530304:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828649.154498:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828649.154503:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828649/real 1787828649] req@ffff9e680a2d6d00 x1874668790530304/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828649.154510:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790530304/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828649.154515:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828649.154516:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828649.154518:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790530304:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828649.665770:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828649.857864:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828650.689815:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828650.881828:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828651.713841:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828651.905887:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828652.737818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828652.929862:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828653.761792:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828653.953929:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828654.273793:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828654.273805:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828654.273809:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828654.273817:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5290 00000100:00080000:3.0:1787828654.273826:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828654.273832:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828654.273888:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790530432/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828654.273911:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828654.273914:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828654.273916:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828654.273919:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5290 00000100:00080000:3.0:1787828654.273922:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828654.273925:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828654.273933:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828654.273944:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790530560/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828654.273957:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828654.273962:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828654.273964:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828654.273990:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5290 00000100:00100000:2.0:1787828654.273991:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790530432:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828654.273999:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828654.274009:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828654.274017:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828654.274055:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790530688/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828654.274095:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828654.274115:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828654.274117:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828654.274120:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5290 00000100:00080000:3.0:1787828654.274123:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828654.274126:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787828654.274131:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787828654.274132:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828654.274144:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790530816/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828654.274151:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828654.274154:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828654.274155:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828654.274157:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5290 00000100:00080000:3.0:1787828654.274160:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828654.274163:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828654.274184:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828654.274192:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828654/real 1787828654] req@ffff9e680a2d4f00 x1874668790530432/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828654.274197:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790530944/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828654.274215:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828654.274216:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790530432/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828654.274222:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828654.274227:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828654.274237:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5290 00000100:00080000:2.0:1787828654.274239:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828654.274242:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828654.274247:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828654.274251:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828654.274273:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790530432:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787828654.274275:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828654.274288:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790530560:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828654.274294:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790531072/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828654.274303:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828654.274306:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828654.274308:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828654.274311:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5290 00000800:00000100:2.0:1787828654.274339:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828654.274346:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828654.274356:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828654.274364:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828654.274368:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828654/real 1787828654] req@ffff9e680a2d61c0 x1874668790530560/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828654.274385:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790531200/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828654.274408:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790530560/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828654.274423:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828654.274425:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828654.274427:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828654.274430:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828654.274432:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828654.274436:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5290 00000100:00100000:2.0:1787828654.274438:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790530560:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828654.274440:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828654.274447:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828654.274450:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828654.274481:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790531328/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828654.274495:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790530688:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828654.274535:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828654.274538:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828654.274540:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828654.274545:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5290 00000100:00080000:3.0:1787828654.274549:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828654.274554:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828654.274558:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828654.274564:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828654.274574:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828654/real 1787828654] req@ffff9e680a2d4b40 x1874668790530688/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828654.274578:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790531456/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828654.274621:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790530688/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828654.274628:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828654.274630:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828654.274633:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790530688:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828654.274638:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790530816:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828654.274651:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828654.274674:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828654/real 1787828654] req@ffff9e680a2d7480 x1874668790530816/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828654.274683:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790530816/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828654.274689:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828654.274691:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828654.274693:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790530816:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828654.274697:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790530944:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828654.274709:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828654.274716:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828654/real 1787828654] req@ffff9e680a2d5e00 x1874668790530944/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828654.274725:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790530944/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828654.274731:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828654.274733:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828654.274735:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790530944:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828654.274739:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790531072:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828654.274750:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828654.274757:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828654/real 1787828654] req@ffff9e680a2d6d00 x1874668790531072/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828654.274764:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790531072/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828654.274772:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828654.274774:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828654.274776:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790531072:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828654.274795:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790531200:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828654.274806:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828654.274811:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828654/real 1787828654] req@ffff9e680a2d7c00 x1874668790531200/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828654.274818:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790531200/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828654.274823:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828654.274825:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828654.274827:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790531200:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828654.274839:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790531328:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828654.274894:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828654.274905:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828654/real 1787828654] req@ffff9e680a2d70c0 x1874668790531328/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828654.274914:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790531328/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828654.274921:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828654.274923:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828654.274926:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790531328:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828654.274931:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790531456:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828654.274945:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828654.274972:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828654/real 1787828654] req@ffff9e680a2d52c0 x1874668790531456/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828654.274985:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790531456/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828654.274994:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828654.274997:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828654.275000:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790531456:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828654.785805:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828654.977844:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828655.809782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828656.001873:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828656.833786:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828657.026352:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828657.857792:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828658.049827:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828658.881911:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828659.073824:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828659.393897:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828659.393907:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828659.393912:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828659.393922:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5296 00000100:00080000:0.0:1787828659.393934:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828659.393949:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828659.393985:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790531584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828659.394011:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828659.394015:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828659.394017:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828659.394021:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5296 00000100:00080000:0.0:1787828659.394023:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828659.394039:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828659.394047:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828659.394077:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790531712/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828659.394087:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828659.394094:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787828659.394094:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790531584:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828659.394098:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828659.394106:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5296 00000100:00080000:0.0:1787828659.394110:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828659.394113:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828659.394135:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828659.394146:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790531840/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828659.394157:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828659.394161:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828659.394162:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828659.394166:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5296 00000100:00080000:0.0:1787828659.394171:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828659.394174:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828659.394179:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828659.394184:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828659.394194:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790531968/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828659.394203:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828659.394206:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828659.394208:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828659.394211:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5296 00000100:00080000:0.0:1787828659.394215:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828659.394219:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828659.394224:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828659.394228:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828659/real 1787828659] req@ffff9e6803e98b40 x1874668790531584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828659.394238:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790532096/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828659.394249:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828659.394252:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790531584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828659.394253:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828659.394257:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828659.394268:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5296 00000100:00080000:0.0:1787828659.394273:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828659.394274:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828659.394279:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787828659.394282:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:0.0:1787828659.394285:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828659.394289:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790531584:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787828659.394300:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790532224/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828659.394327:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828659.394330:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828659.394331:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828659.394338:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5296 00000100:00100000:1.0:1787828659.394339:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790531712:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828659.394341:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828659.394346:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828659.394351:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828659.394362:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790532352/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828659.394370:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828659.394372:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828659.394374:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828659.394377:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828659.394378:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5296 00000100:00080000:0.0:1787828659.394384:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828659.394388:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828659.394392:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828659.394397:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828659/real 1787828659] req@ffff9e6803e98780 x1874668790531712/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828659.394403:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790532480/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828659.394417:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828659.394419:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790531712/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828659.394422:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828659.394426:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828659.394436:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5296 00000100:00080000:0.0:1787828659.394440:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828659.394440:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828659.394449:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787828659.394450:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:0.0:1787828659.394458:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828659.394458:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790531712:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828659.394469:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790531840:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828659.394474:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790532608/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828659.394492:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828659.394508:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828659/real 1787828659] req@ffff9e6803e9bc00 x1874668790531840/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828659.394521:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790531840/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828659.394529:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828659.394533:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828659.394538:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790531840:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828659.394546:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790531968:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828659.394561:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828659.394570:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828659/real 1787828659] req@ffff9e6803e9b840 x1874668790531968/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828659.394578:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790531968/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828659.394584:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828659.394586:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828659.394588:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790531968:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828659.394592:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790532096:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828659.394606:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828659.394617:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828659/real 1787828659] req@ffff9e6803e9a1c0 x1874668790532096/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828659.394650:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790532096/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828659.394662:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828659.394667:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828659.394671:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790532096:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828659.394679:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790532224:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828659.394701:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828659.394754:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828659/real 1787828659] req@ffff9e6803e9b480 x1874668790532224/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828659.394766:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790532224/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828659.394775:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828659.394778:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828659.394781:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790532224:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828659.394823:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790532352:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828659.394848:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828659.394863:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828659/real 1787828659] req@ffff9e6803e9a580 x1874668790532352/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828659.394879:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790532352/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828659.394889:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828659.394893:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828659.394898:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790532352:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828659.394907:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790532480:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828659.394934:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828659.394949:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828659/real 1787828659] req@ffff9e6803e99680 x1874668790532480/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828659.394966:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790532480/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828659.394977:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828659.394982:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828659.394986:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790532480:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828659.394995:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790532608:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828659.395021:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828659.395068:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828659/real 1787828659] req@ffff9e6803e992c0 x1874668790532608/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828659.395082:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790532608/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828659.395091:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828659.395095:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828659.395099:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790532608:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828659.905786:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828660.097941:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828660.929930:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:0.0:1787828661.121940:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828661.953807:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828662.145838:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828662.977801:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828663.169858:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828664.001774:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828664.193896:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828664.513790:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828664.513799:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828664.513803:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828664.513811:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5301 00000100:00080000:1.0:1787828664.513820:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828664.513826:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828664.513857:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790532736/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828664.513881:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828664.513884:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828664.513885:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828664.513889:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5301 00000100:00080000:1.0:1787828664.513891:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828664.513894:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828664.513902:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828664.513910:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790532864/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828664.513918:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828664.513923:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828664.513925:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828664.513925:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790532736:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828664.513933:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5301 00000100:00080000:1.0:1787828664.513936:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828664.513938:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828664.513941:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828664.513951:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790532992/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828664.513958:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828664.513961:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828664.513962:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828664.513965:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5301 00000100:00080000:1.0:1787828664.513968:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828664.513970:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828664.513973:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828664.513981:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790533120/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828664.514005:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828664.514007:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828664.514011:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828664.514012:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828664.514016:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5301 00000100:00080000:1.0:1787828664.514020:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828664.514024:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828664.514028:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828664.514038:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790533248/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828664.514045:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828664.514046:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828664/real 1787828664] req@ffff9e6831aa8f00 x1874668790532736/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828664.514049:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828664.514053:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828664.514062:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5301 00000100:00080000:1.0:1787828664.514085:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828664.514094:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828664.514102:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828664.514111:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790532736/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828664.514117:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790533376/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828664.514154:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828664.514160:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828664.514161:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828664.514168:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790532736:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828664.514169:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828664.514175:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828664.514182:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5301 00000100:00080000:1.0:1787828664.514187:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828664.514192:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828664.514194:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790532864:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787828664.514202:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828664.514223:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790533504/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828664.514232:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828664.514236:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828664.514240:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828664.514242:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828664.514246:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828664/real 1787828664] req@ffff9e6831aaa580 x1874668790532864/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828664.514248:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5301 00000100:00080000:1.0:1787828664.514260:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828664.514263:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790532864/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828664.514267:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828664.514275:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828664.514278:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828664.514283:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828664.514288:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790533632/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828664.514288:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790532864:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828664.514301:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790532992:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828664.514304:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828664.514308:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828664.514309:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828664.514313:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5301 00000100:00080000:1.0:1787828664.514317:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828664.514319:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787828664.514324:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1787828664.514327:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828664.514335:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828664/real 1787828664] req@ffff9e6831aa9680 x1874668790532992/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828664.514340:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790533760/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828664.514357:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790532992/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828664.514368:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828664.514371:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828664.514374:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790532992:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828664.514378:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790533120:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828664.514390:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828664.514396:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828664/real 1787828664] req@ffff9e6831aa9e00 x1874668790533120/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828664.514403:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790533120/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828664.514408:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828664.514410:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828664.514412:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790533120:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828664.514415:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790533248:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828664.514425:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828664.514431:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828664/real 1787828664] req@ffff9e6831aa8b40 x1874668790533248/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828664.514445:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790533248/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828664.514450:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828664.514452:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828664.514454:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790533248:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828664.514458:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790533376:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828664.514469:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828664.514475:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828664/real 1787828664] req@ffff9e6831aaa940 x1874668790533376/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828664.514482:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790533376/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828664.514488:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828664.514490:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828664.514491:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790533376:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828664.514509:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790533504:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828664.514521:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828664.514527:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828664/real 1787828664] req@ffff9e6831aa8780 x1874668790533504/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828664.514535:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790533504/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828664.514540:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828664.514542:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828664.514543:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790533504:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828664.514547:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790533632:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828664.514557:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828664.514563:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828664/real 1787828664] req@ffff9e6831aaa1c0 x1874668790533632/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828664.514570:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790533632/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828664.514575:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828664.514577:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828664.514579:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790533632:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828664.514582:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790533760:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828664.514591:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828664.514597:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828664/real 1787828664] req@ffff9e6831aa8000 x1874668790533760/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828664.514603:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790533760/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828664.514608:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828664.514610:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828664.514612:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790533760:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828665.025782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828665.217901:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828666.049780:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828666.241853:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828667.073863:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828667.266013:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828668.097844:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828668.289874:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828669.121800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828669.313904:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828669.633872:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828669.633881:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828669.633884:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828669.633893:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5306 00000100:00080000:3.0:1787828669.633918:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828669.633923:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828669.633972:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790533888/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828669.634010:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828669.634013:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828669.634015:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828669.634019:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5306 00000100:00080000:3.0:1787828669.634022:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828669.634026:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828669.634035:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828669.634036:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790533888:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828669.634049:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790534016/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828669.634061:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828669.634065:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828669.634067:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828669.634072:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5306 00000100:00080000:3.0:1787828669.634089:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828669.634094:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828669.634100:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828669.634106:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828669.634116:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790534144/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828669.634128:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828669.634132:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828669.634134:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828669.634138:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5306 00000100:00080000:3.0:1787828669.634142:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828669.634147:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828669.634152:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828669.634158:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828669/real 1787828669] req@ffff9e680a2d4000 x1874668790533888/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828669.634168:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790534272/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828669.634197:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828669.634202:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828669.634204:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828669.634209:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790533888/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828669.634210:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5306 00000100:00080000:3.0:1787828669.634216:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828669.634225:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787828669.634231:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828669.634232:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828669.634240:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828669.634248:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790534400/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828669.634249:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790533888:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828669.634265:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828669.634269:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828669.634271:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828669.634276:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5306 00000100:00080000:3.0:1787828669.634280:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828669.634281:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790534016:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828669.634287:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828669.634294:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828669.634308:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790534528/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828669.634317:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828669.634332:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828669.634336:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828669/real 1787828669] req@ffff9e680a2d6580 x1874668790534016/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828669.634337:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828669.634341:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828669.634349:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5306 00000100:00080000:3.0:1787828669.634355:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828669.634361:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828669.634365:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790534016/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828669.634379:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828669.634394:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828669.634396:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790534656/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828669.634402:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828669.634412:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790534016:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828669.634416:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828669.634422:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787828669.634425:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790534144:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828669.634426:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828669.634436:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5306 00000100:00080000:3.0:1787828669.634450:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828669.634454:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828669.634460:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828669.634464:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828669.634475:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790534784/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828669.634477:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828669/real 1787828669] req@ffff9e680a2d43c0 x1874668790534144/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828669.634516:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790534144/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828669.634517:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828669.634526:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828669.634531:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828669.634540:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5306 00000100:00080000:2.0:1787828669.634546:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828669.634548:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828669.634555:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787828669.634557:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:3.0:1787828669.634562:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828669.634564:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790534144:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828669.634585:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790534912/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828669.634596:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790534272:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828669.634617:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828669.634625:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828669/real 1787828669] req@ffff9e680a2d5a40 x1874668790534272/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828669.634633:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790534272/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828669.634647:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828669.634649:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828669.634651:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790534272:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828669.634654:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790534400:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828669.634664:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828669.634669:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828669/real 1787828669] req@ffff9e680a2d4780 x1874668790534400/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828669.634676:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790534400/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828669.634681:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828669.634683:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828669.634685:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790534400:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828669.634716:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790534528:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828669.634729:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828669.634735:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828669/real 1787828669] req@ffff9e680a2d4000 x1874668790534528/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828669.634743:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790534528/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828669.634749:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828669.634751:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828669.634753:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790534528:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828669.634757:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790534656:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828669.634766:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828669.634772:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828669/real 1787828669] req@ffff9e680a2d52c0 x1874668790534656/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828669.634790:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790534656/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828669.634795:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828669.634797:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828669.634799:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790534656:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828669.634802:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790534784:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828669.634811:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828669.634817:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828669/real 1787828669] req@ffff9e680a2d70c0 x1874668790534784/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828669.634824:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790534784/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828669.634830:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828669.634831:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828669.634833:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790534784:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828669.634836:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790534912:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828669.634845:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828669.634851:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828669/real 1787828669] req@ffff9e680a2d7c00 x1874668790534912/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828669.634873:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790534912/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828669.634881:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828669.634884:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828669.634888:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790534912:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828670.145804:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828670.337853:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828671.169835:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828671.361884:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828672.193761:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828672.385859:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828673.217792:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828673.409829:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828674.241753:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828674.433863:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828674.753837:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828674.753845:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828674.753848:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828674.753857:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5311 00000100:00080000:0.0:1787828674.753866:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828674.753872:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828674.753904:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790535040/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828674.753940:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828674.753943:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828674.753945:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828674.753948:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5311 00000100:00080000:0.0:1787828674.753950:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828674.753953:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828674.753960:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828674.753970:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790535168/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828674.753977:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828674.753980:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828674.753983:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828674.753985:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790535040:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828674.753987:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5311 00000100:00080000:0.0:1787828674.753993:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828674.753997:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828674.754000:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828674.754009:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790535296/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828674.754015:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828674.754017:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828674.754019:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828674.754021:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5311 00000100:00080000:0.0:1787828674.754024:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828674.754027:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828674.754030:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828674.754037:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790535424/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828674.754042:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828674.754044:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828674.754045:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828674.754050:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828674.754052:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5311 00000100:00080000:0.0:1787828674.754057:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828674.754059:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828674.754063:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828674.754072:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790535552/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828674.754078:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828674.754081:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828674.754082:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828674.754085:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5311 00000100:00080000:0.0:1787828674.754087:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828674.754090:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828674/real 1787828674] req@ffff9e6803e9a940 x1874668790535040/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828674.754093:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828674.754100:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828674.754113:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790535680/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828674.754117:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790535040/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828674.754147:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828674.754148:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828674.754153:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828674.754154:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787828674.754155:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828674.754161:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5311 00000100:00100000:2.0:1787828674.754162:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790535040:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828674.754165:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828674.754169:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828674.754172:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828674.754179:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790535808/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828674.754185:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790535168:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828674.754188:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828674.754192:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828674.754194:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828674.754197:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5311 00000100:00080000:0.0:1787828674.754201:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828674.754203:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828674.754207:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828674.754210:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828674.754220:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790535936/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828674.754225:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828674/real 1787828674] req@ffff9e6803e99e00 x1874668790535168/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828674.754236:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790535168/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828674.754241:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828674.754248:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828674.754250:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828674.754255:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828674.754256:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5311 00000100:00080000:0.0:1787828674.754261:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828674.754262:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787828674.754266:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828674.754269:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790535168:192.168.203.109@tcp:38:kworker.0 00000008:00100000:0.0:1787828674.754272:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828674.754277:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790535296:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828674.754282:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790536064/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828674.754297:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828674.754307:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828674/real 1787828674] req@ffff9e6803e98f00 x1874668790535296/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828674.754315:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790535296/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828674.754320:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828674.754322:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828674.754324:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790535296:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828674.754328:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790535424:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828674.754339:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828674.754344:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828674/real 1787828674] req@ffff9e6803e9ad00 x1874668790535424/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828674.754351:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790535424/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828674.754357:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828674.754359:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828674.754361:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790535424:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828674.754364:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790535552:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828674.754374:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828674.754379:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828674/real 1787828674] req@ffff9e6803e992c0 x1874668790535552/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828674.754386:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790535552/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828674.754391:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828674.754393:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828674.754395:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790535552:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828674.754398:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790535680:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828674.754408:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828674.754413:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828674/real 1787828674] req@ffff9e6803e99680 x1874668790535680/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828674.754421:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790535680/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828674.754427:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828674.754428:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828674.754430:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790535680:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828674.754444:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790535808:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828674.754463:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828674.754475:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828674/real 1787828674] req@ffff9e6803e9a580 x1874668790535808/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828674.754488:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790535808/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828674.754497:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828674.754500:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828674.754502:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790535808:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828674.754506:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790535936:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828674.754521:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828674.754529:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828674/real 1787828674] req@ffff9e6803e9b480 x1874668790535936/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828674.754536:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790535936/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828674.754545:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828674.754547:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828674.754549:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790535936:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828674.754553:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790536064:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828674.754562:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828674.754568:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828674/real 1787828674] req@ffff9e6803e9a1c0 x1874668790536064/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828674.754574:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790536064/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828674.754579:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828674.754581:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828674.754582:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790536064:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828675.265815:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828675.457895:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828676.289805:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828676.481851:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828677.313800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828677.505876:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828678.337971:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828678.529886:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828679.361845:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828679.553842:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828679.873786:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828679.873794:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828679.873798:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828679.873806:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5316 00000100:00080000:1.0:1787828679.873815:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828679.873820:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828679.873852:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790536192/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828679.873874:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828679.873876:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828679.873878:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828679.873881:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5316 00000100:00080000:1.0:1787828679.873898:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828679.873902:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828679.873910:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828679.873915:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790536192:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828679.873924:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790536320/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828679.873934:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828679.873936:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828679.873938:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828679.873940:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5316 00000100:00080000:1.0:1787828679.873943:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828679.873946:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828679.873949:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828679.873957:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790536448/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828679.873965:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828679.873968:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828679.873969:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828679.873972:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5316 00000100:00080000:1.0:1787828679.873975:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828679.873981:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828679.873982:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828679.873989:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828679.873997:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790536576/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828679.874004:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828679.874007:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828679.874008:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828679.874010:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5316 00000100:00080000:1.0:1787828679.874013:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828679.874016:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828679.874019:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828679.874026:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828679/real 1787828679] req@ffff9e6831aaad00 x1874668790536192/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828679.874030:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790536704/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828679.874049:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828679.874052:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828679.874053:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828679.874056:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790536192/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828679.874059:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5316 00000100:00080000:1.0:1787828679.874064:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828679.874069:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828679.874076:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828679.874078:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828679.874084:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828679.874086:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790536832/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828679.874090:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790536192:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828679.874103:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828679.874105:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828679.874107:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828679.874110:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5316 00000100:00080000:1.0:1787828679.874112:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828679.874114:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828679.874118:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828679.874118:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790536320:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828679.874129:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790536960/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828679.874138:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828679.874141:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828679.874142:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828679.874147:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828679.874151:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5316 00000100:00080000:1.0:1787828679.874157:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828679.874161:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828679/real 1787828679] req@ffff9e6831aabc00 x1874668790536320/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828679.874166:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828679.874175:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828679.874181:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790536320/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828679.874190:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828679.874194:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790537088/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828679.874197:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828679.874207:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790536320:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828679.874210:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828679.874216:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828679.874217:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828679.874222:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790536448:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828679.874224:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5316 00000100:00080000:1.0:1787828679.874230:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828679.874234:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828679.874238:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828679.874249:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790537216/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828679.874257:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828679.874274:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828679/real 1787828679] req@ffff9e6831aab480 x1874668790536448/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828679.874288:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790536448/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828679.874297:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828679.874301:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828679.874304:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790536448:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828679.874311:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790536576:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828679.874332:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828679.874340:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828679/real 1787828679] req@ffff9e6831aa9a40 x1874668790536576/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828679.874349:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790536576/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828679.874355:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828679.874357:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828679.874359:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790536576:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828679.874363:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790536704:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828679.874374:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828679.874379:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828679/real 1787828679] req@ffff9e6831aa8000 x1874668790536704/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828679.874386:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790536704/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828679.874391:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828679.874393:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828679.874395:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790536704:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828679.874398:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790536832:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828679.874412:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828679.874422:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828679/real 1787828679] req@ffff9e6831aaa1c0 x1874668790536832/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828679.874434:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790536832/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828679.874451:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828679.874454:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828679.874457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790536832:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828679.874470:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790536960:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828679.874486:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828679.874493:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828679/real 1787828679] req@ffff9e6831aa8780 x1874668790536960/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828679.874500:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790536960/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828679.874505:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828679.874507:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828679.874509:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790536960:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828679.874515:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790537088:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828679.874533:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828679.874544:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828679/real 1787828679] req@ffff9e6831aaa940 x1874668790537088/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828679.874557:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790537088/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828679.874566:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828679.874569:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828679.874573:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790537088:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828679.874579:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790537216:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828679.874599:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828679.874610:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828679/real 1787828679] req@ffff9e6831aa8b40 x1874668790537216/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828679.874619:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790537216/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828704 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828679.874628:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828679.874631:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828679.874634:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790537216:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828680.385789:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828680.577896:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828681.409771:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828681.601883:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828682.433844:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828682.625842:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828683.457839:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828683.649908:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828684.481892:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828684.673842:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828684.993812:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828684.993823:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828684.993828:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828684.993838:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5321 00000100:00080000:1.0:1787828684.993851:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828684.993890:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828684.993924:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790537344/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828684.993966:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828684.993970:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828684.993973:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828684.993987:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5321 00000100:00080000:1.0:1787828684.993993:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828684.993999:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828684.994025:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828684.994026:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790537344:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828684.994044:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790537472/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828684.994058:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828684.994063:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828684.994065:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828684.994069:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5321 00000100:00080000:1.0:1787828684.994076:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828684.994081:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787828684.994086:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1787828684.994090:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828684.994104:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790537600/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828684.994111:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828684.994114:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828684.994115:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828684.994119:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5321 00000100:00080000:1.0:1787828684.994122:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828684.994125:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828684/real 1787828684] req@ffff9e6831aa9e00 x1874668790537344/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828684.994128:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828684.994134:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828684.994147:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790537728/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828684.994150:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790537344/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828684.994165:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828684.994185:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828684.994187:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828684.994187:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828684.994197:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828684.994198:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5321 00000100:00080000:1.0:1787828684.994204:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828684.994206:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790537344:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828684.994210:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828684.994216:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828684.994226:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790537856/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828684.994234:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828684.994235:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790537472:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828684.994238:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828684.994242:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828684.994247:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5321 00000100:00080000:1.0:1787828684.994250:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828684.994253:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828684.994273:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828684.994282:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828684.994285:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790537984/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828684.994292:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828684.994297:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828684.994297:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828684/real 1787828684] req@ffff9e6831aa9680 x1874668790537472/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828684.994299:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828684.994305:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5321 00000100:00080000:1.0:1787828684.994311:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828684.994315:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828684.994318:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790537472/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828684.994320:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828684.994347:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828684.994350:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828684.994353:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790537472:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828684.994358:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790538112/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828684.994361:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790537600:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828684.994380:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828684.994382:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828684.994384:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828684.994388:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828684.994408:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5321 00000100:00080000:1.0:1787828684.994413:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828684.994416:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828684.994421:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828684/real 1787828684] req@ffff9e6831aaa580 x1874668790537600/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828684.994422:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828684.994435:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790538240/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828684.994438:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790537600/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828684.994456:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828684.994474:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828684.994478:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828684.994481:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828684.994483:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828684.994488:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790537600:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828684.994490:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5321 00000100:00080000:1.0:1787828684.994497:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828684.994501:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790537728:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787828684.994502:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828684.994509:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828684.994522:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790538368/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828684.994546:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828684.994557:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828684/real 1787828684] req@ffff9e6831aa8f00 x1874668790537728/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828684.994566:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790537728/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828684.994572:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828684.994574:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828684.994576:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790537728:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828684.994586:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790537856:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828684.994596:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828684.994601:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828684/real 1787828684] req@ffff9e6831aa8b40 x1874668790537856/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828684.994608:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790537856/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828684.994614:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828684.994616:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828684.994618:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790537856:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828684.994621:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790537984:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828684.994630:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828684.994635:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828684/real 1787828684] req@ffff9e6831aaa940 x1874668790537984/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828684.994651:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790537984/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828684.994657:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828684.994659:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828684.994662:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790537984:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828684.994666:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790538112:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828684.994677:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828684.994683:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828684/real 1787828684] req@ffff9e6831aa8780 x1874668790538112/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828684.994690:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790538112/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828684.994744:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828684.994750:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828684.994755:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790538112:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828684.994766:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790538240:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828684.994802:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828684.994821:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828684/real 1787828684] req@ffff9e6831aaa1c0 x1874668790538240/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828684.994835:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790538240/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828684.994844:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828684.994888:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828684.994891:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790538240:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828684.994895:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790538368:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828684.994910:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828684.994918:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828684/real 1787828684] req@ffff9e6831aa8000 x1874668790538368/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828684.994925:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790538368/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828709 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828684.994931:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828684.994932:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828684.994934:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790538368:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828685.505798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828685.697914:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828686.529998:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828686.722024:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828687.553872:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828687.745870:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828688.577806:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828688.769888:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828689.601795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828689.793890:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828690.113809:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828690.113830:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828690.113861:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828690.113869:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5326 00000100:00080000:0.0:1787828690.113879:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828690.113885:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828690.113918:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790538496/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828690.113941:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828690.113944:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828690.113945:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828690.113948:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5326 00000100:00080000:0.0:1787828690.113951:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828690.113954:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828690.113962:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828690.113975:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790538624/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828690.113976:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790538496:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828690.113992:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828690.113997:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828690.113998:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828690.114002:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5326 00000100:00080000:0.0:1787828690.114004:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828690.114007:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828690.114010:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828690.114018:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790538752/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828690.114025:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828690.114027:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828690.114029:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828690.114031:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5326 00000100:00080000:0.0:1787828690.114034:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828690.114036:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828690.114040:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828690.114044:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828690.114049:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790538880/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828690.114057:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828690.114060:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828690.114062:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828690.114065:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5326 00000100:00080000:0.0:1787828690.114067:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828690.114069:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828690.114073:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828690.114095:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790539008/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828690.114095:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828690/real 1787828690] req@ffff9e6803e9b840 x1874668790538496/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828690.114123:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828690.114126:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828690.114128:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828690.114131:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5326 00000100:00100000:2.0:1787828690.114135:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790538496/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828690.114137:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828690.114144:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828690.114152:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828690.114155:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828690.114162:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828690.114163:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790539136/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828690.114168:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790538496:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828690.114181:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828690.114183:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828690.114185:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828690.114188:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5326 00000100:00080000:0.0:1787828690.114190:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828690.114206:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790538624:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828690.114213:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828690.114217:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828690.114226:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790539264/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828690.114233:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828690.114240:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828690.114243:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828690.114245:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828690.114249:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5326 00000100:00080000:0.0:1787828690.114252:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828690.114257:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828690.114258:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828690/real 1787828690] req@ffff9e6803e9bc00 x1874668790538624/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828690.114272:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828690.114284:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790539392/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828690.114321:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828690.114324:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790538624/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828690.114336:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828690.114339:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828690.114344:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5326 00000100:00080000:0.0:1787828690.114351:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828690.114360:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787828690.114360:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828690.114367:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:0.0:1787828690.114368:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828690.114371:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790538624:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828690.114379:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790538752:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828690.114388:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790539520/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828690.114411:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828690.114427:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828690/real 1787828690] req@ffff9e6803e98780 x1874668790538752/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828690.114441:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790538752/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828690.114450:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828690.114454:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828690.114457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790538752:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828690.114464:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790538880:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828690.114481:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828690.114488:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828690/real 1787828690] req@ffff9e6803e98b40 x1874668790538880/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828690.114496:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790538880/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828690.114515:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828690.114517:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828690.114519:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790538880:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828690.114524:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790539008:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828690.114535:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828690.114542:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828690/real 1787828690] req@ffff9e6803e9a1c0 x1874668790539008/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828690.114549:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790539008/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828690.114554:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828690.114557:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828690.114558:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790539008:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828690.114562:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790539136:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828690.114571:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828690.114577:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828690/real 1787828690] req@ffff9e6803e9b480 x1874668790539136/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828690.114584:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790539136/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828690.114590:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828690.114592:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828690.114595:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790539136:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828690.114619:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790539264:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828690.114629:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828690.114635:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828690/real 1787828690] req@ffff9e6803e9a580 x1874668790539264/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828690.114641:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790539264/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828690.114647:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828690.114648:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828690.114650:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790539264:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828690.114654:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790539392:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828690.114663:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828690.114669:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828690/real 1787828690] req@ffff9e6803e99680 x1874668790539392/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828690.114676:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790539392/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828690.114681:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828690.114683:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828690.114684:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790539392:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828690.114687:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790539520:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828690.114716:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828690.114724:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828690/real 1787828690] req@ffff9e6803e992c0 x1874668790539520/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828690.114731:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790539520/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828690.114737:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828690.114739:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828690.114741:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790539520:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828690.625865:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828690.817929:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828691.649844:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828691.841886:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828692.673781:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828692.865943:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828693.697778:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828693.889878:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828694.721813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828694.913904:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828695.233856:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828695.233886:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828695.233892:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828695.233905:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5331 00000100:00080000:0.0:1787828695.233919:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828695.233929:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828695.234008:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790539648/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828695.234059:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828695.234065:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828695.234069:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828695.234076:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5331 00000100:00080000:0.0:1787828695.234083:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828695.234090:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828695.234114:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828695.234121:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790539648:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787828695.234138:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790539776/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828695.234153:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828695.234158:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828695.234161:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828695.234166:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5331 00000100:00080000:0.0:1787828695.234187:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828695.234193:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1787828695.234197:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787828695.234201:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828695.234230:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790539904/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828695.234255:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828695.234260:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787828695.234261:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828695/real 1787828695] req@ffff9e6803e9ad00 x1874668790539648/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828695.234263:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828695.234268:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5331 00000100:00080000:0.0:1787828695.234274:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828695.234279:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828695.234284:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790539648/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828695.234285:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828695.234299:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790540032/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828695.234300:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828695.234305:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828695.234312:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790539648:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828695.234313:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828695.234317:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828695.234320:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828695.234325:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5331 00000100:00080000:0.0:1787828695.234329:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828695.234333:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828695.234338:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828695.234341:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790539776:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828695.234351:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790540160/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828695.234373:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828695.234381:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828695.234386:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828695.234389:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828695.234391:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828695/real 1787828695] req@ffff9e6803e98f00 x1874668790539776/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828695.234394:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5331 00000100:00080000:0.0:1787828695.234399:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828695.234405:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828695.234408:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790539776/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828695.234412:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787828695.234423:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828695.234427:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790540288/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828695.234428:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828695.234433:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790539776:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828695.234443:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790539904:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828695.234449:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828695.234464:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828695.234467:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828695.234473:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5331 00000800:00000100:1.0:1787828695.234473:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828695.234478:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828695.234484:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828695.234491:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828695.234492:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828695/real 1787828695] req@ffff9e6803e99e00 x1874668790539904/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828695.234507:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790540416/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828695.234511:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790539904/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828695.234521:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828695.234523:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828695.234526:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828695.234529:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828695.234530:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787828695.234535:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5331 00000100:00100000:1.0:1787828695.234536:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790539904:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828695.234541:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828695.234547:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828695.234547:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790540032:192.168.203.109@tcp:8:kworker.0 00000008:00100000:0.0:1787828695.234555:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828695.234581:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790540544/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828695.234590:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828695.234595:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828695.234600:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828695.234603:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828695.234607:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828695/real 1787828695] req@ffff9e6803e9a940 x1874668790540032/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828695.234609:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5331 00000100:00080000:0.0:1787828695.234615:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828695.234621:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828695.234625:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790540032/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828695.234628:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828695.234638:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828695.234643:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790540672/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828695.234643:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828695.234648:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790540032:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828695.234694:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790540160:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828695.234757:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828695.234795:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828695/real 1787828695] req@ffff9e6803e9ad00 x1874668790540160/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828695.234826:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790540160/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828695.234839:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828695.234842:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828695.234845:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790540160:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828695.234851:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790540288:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828695.234890:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828695.234902:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828695/real 1787828695] req@ffff9e6803e992c0 x1874668790540288/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828695.234913:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790540288/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828695.234922:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828695.234924:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828695.234927:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790540288:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828695.234932:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790540416:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828695.234947:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828695.234965:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828695/real 1787828695] req@ffff9e6803e99680 x1874668790540416/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828695.234992:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790540416/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828695.234999:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828695.235001:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828695.235005:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790540416:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828695.235010:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790540544:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828695.235036:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828695.235044:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828695/real 1787828695] req@ffff9e6803e9a580 x1874668790540544/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828695.235053:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790540544/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828695.235060:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828695.235062:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828695.235065:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790540544:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828695.235070:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790540672:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828695.235082:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828695.235090:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828695/real 1787828695] req@ffff9e6803e9b480 x1874668790540672/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828695.235098:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790540672/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828695.235109:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828695.235112:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828695.235115:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790540672:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828695.745848:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828695.937940:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828696.769800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828696.961879:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828697.793851:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828697.985917:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828698.817862:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828699.009861:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828699.841881:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828700.033978:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828700.353845:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828700.353853:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828700.353856:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828700.353864:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5337 00000100:00080000:1.0:1787828700.353873:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828700.353878:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828700.353910:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790540800/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828700.353934:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828700.353936:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828700.353938:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828700.353940:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5337 00000100:00080000:1.0:1787828700.353943:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828700.353946:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828700.353956:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828700.353965:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790540800:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828700.353972:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790540928/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828700.353985:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828700.353988:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828700.353991:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828700.353994:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5337 00000100:00080000:1.0:1787828700.353999:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828700.354003:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828700.354009:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828700.354020:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790541056/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828700.354030:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828700.354033:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828700.354035:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828700.354040:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5337 00000100:00080000:1.0:1787828700.354044:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828700.354048:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828700.354054:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828700.354059:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790540928:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828700.354069:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790541184/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828700.354081:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828700.354084:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828700.354086:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828700.354087:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790541056:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828700.354092:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5337 00000100:00080000:1.0:1787828700.354097:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828700.354101:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828700.354106:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828700.354113:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790541184:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828700.354119:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790541312/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828700.354127:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828700.354130:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828700.354131:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828700.354133:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790541312:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787828700.354144:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5337 00000100:00080000:1.0:1787828700.354149:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828700.354152:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828700.354156:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828700.354165:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790541440/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828700.354176:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828700.354178:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828700.354180:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828700.354184:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5337 00000100:00080000:1.0:1787828700.354188:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828700.354191:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790541440:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828700.354196:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828700.354204:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828700.354218:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790541568/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828700.354231:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828700.354235:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828700.354238:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828700.354243:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5337 00000100:00100000:2.0:1787828700.354244:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790541568:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828700.354252:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828700.354258:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828700.354262:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828700.354274:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790541696/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828700.354285:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828700.354288:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828700.354289:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828700.354293:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5337 00000100:00080000:1.0:1787828700.354296:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828700.354299:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790541696:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787828700.354301:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828700.354308:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828700.354319:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790541824/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828700.354334:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790541824:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828700.865808:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828701.057883:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828701.889874:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828702.081872:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828702.913818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828703.105851:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787828703.425968:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:3.0:1787828703.426010:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787828703.426028:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787828703.426082:0:2386:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:3.0:1787828703.426102:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787828703.426117:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828700/real 1787828703] req@ffff9e6831aa8b40 x1874668790541824/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787828703.426131:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787828703.426170:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787828703.426187:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790541824/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787828703.426197:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787828703.426222:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00080000:2.0:1787828703.426229:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000800:00000100:3.0:1787828703.426240:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00080000:2.0:1787828703.426245:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828703.426256:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790541824:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787828703.426264:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787828703.426279:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828700/real 1787828703] req@ffff9e6831aa8780 x1874668790541568/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828703.426304:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790541568/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828703.426316:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828703.426321:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828703.426326:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790541568:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828703.426336:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828700/real 1787828703] req@ffff9e6831aaa1c0 x1874668790541440/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828703.426355:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790541440/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828703.426371:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828703.426376:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828703.426381:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790541440:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828703.426392:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828700/real 1787828703] req@ffff9e6831aa8000 x1874668790541312/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828703.426410:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790541312/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828703.426421:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828703.426426:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828703.426431:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790541312:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828703.426441:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828700/real 1787828703] req@ffff9e6831aaad00 x1874668790541184/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828703.426458:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790541184/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828703.426471:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828703.426475:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828703.426481:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790541184:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828703.426490:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828700/real 1787828703] req@ffff9e6831aab480 x1874668790540928/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828703.426509:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790540928/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828703.426521:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828703.426525:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828703.426530:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790540928:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828703.426540:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828700/real 1787828703] req@ffff9e6831aabc00 x1874668790541056/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828703.426557:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790541056/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828703.426568:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828703.426573:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828703.426577:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790541056:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828703.426585:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828700/real 1787828703] req@ffff9e6831aa9a40 x1874668790540800/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828703.426602:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790540800/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828703.426638:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828703.426644:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828703.426650:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790540800:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787828703.426693:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828700/real 1787828703] req@ffff9e6831aaa940 x1874668790541696/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828703.426779:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790541696/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828703.426794:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828703.426799:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828703.426805:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790541696:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828703.937872:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828704.129897:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828704.961879:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828705.153892:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828705.473771:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828705.473779:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828705.473783:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828705.473791:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5342 00000100:00080000:1.0:1787828705.473800:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828705.473806:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828705.473836:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790541952/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828705.473858:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828705.473861:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828705.473863:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828705.473865:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5342 00000100:00080000:1.0:1787828705.473868:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828705.473872:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828705.473879:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828705.473891:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790542080/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828705.473893:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790541952:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828705.473908:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828705.473912:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828705.473914:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828705.473919:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5342 00000100:00080000:1.0:1787828705.473923:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828705.473927:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828705.473933:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828705.473946:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790542208/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828705.473957:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828705.473961:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828705.473963:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828705.473969:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828705.473972:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5342 00000100:00080000:1.0:1787828705.473981:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828705.473986:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828705.473993:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828705.474007:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790542336/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828705.474018:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828705.474020:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828705.474022:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828705.474024:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828705/real 1787828705] req@ffff9e6831aa8f00 x1874668790541952/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828705.474029:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5342 00000100:00080000:1.0:1787828705.474037:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828705.474045:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828705.474051:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828705.474052:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790541952/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828705.474065:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828705.474071:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828705.474076:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790541952:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828705.474093:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790542464/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828705.474105:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790542080:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828705.474107:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828705.474115:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828705.474117:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828705.474122:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5342 00000100:00080000:1.0:1787828705.474127:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828705.474132:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828705.474138:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828705.474141:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828705.474156:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790542592/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828705.474161:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828705/real 1787828705] req@ffff9e6831aaa580 x1874668790542080/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828705.474178:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828705.474183:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828705.474185:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828705.474188:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790542080/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828705.474193:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5342 00000100:00080000:1.0:1787828705.474202:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828705.474209:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828705.474219:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828705.474221:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828705.474234:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790542720/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828705.474235:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828705.474244:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790542080:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828705.474256:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828705.474258:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790542208:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828705.474262:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828705.474266:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828705.474270:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5342 00000100:00080000:1.0:1787828705.474275:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828705.474279:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828705.474285:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828705.474289:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828705.474303:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790542848/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828705.474309:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828705/real 1787828705] req@ffff9e6831aa9680 x1874668790542208/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828705.474321:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828705.474326:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828705.474329:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828705.474332:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790542208/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828705.474337:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5342 00000100:00080000:1.0:1787828705.474345:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828705.474348:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828705.474352:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787828705.474357:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:1.0:1787828705.474360:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828705.474363:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790542208:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828705.474372:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790542336:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828705.474378:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790542976/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828705.474400:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828705.474415:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828705/real 1787828705] req@ffff9e6831aa9e00 x1874668790542336/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828705.474428:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790542336/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828705.474450:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828705.474453:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828705.474457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790542336:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828705.474475:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790542464:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828705.474497:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828705.474509:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828705/real 1787828705] req@ffff9e6831aaa940 x1874668790542464/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828705.474521:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790542464/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828705.474530:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828705.474533:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828705.474536:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790542464:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828705.474543:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790542592:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828705.474562:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828705.474581:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828705/real 1787828705] req@ffff9e6831aa9a40 x1874668790542592/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828705.474595:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790542592/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828705.474605:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828705.474609:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828705.474613:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790542592:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828705.474619:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790542720:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828705.474641:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828705.474653:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828705/real 1787828705] req@ffff9e6831aabc00 x1874668790542720/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828705.474666:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790542720/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828705.474674:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828705.474678:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828705.474682:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790542720:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828705.474689:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790542848:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828705.474752:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828705.474771:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828705/real 1787828705] req@ffff9e6831aab480 x1874668790542848/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828705.474786:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790542848/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828705.474794:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828705.474797:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828705.474800:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790542848:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828705.474806:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790542976:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828705.474833:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828705.474845:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828705/real 1787828705] req@ffff9e6831aaad00 x1874668790542976/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828705.474856:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790542976/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828705.474864:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828705.474867:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828705.474870:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790542976:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828705.985769:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828706.177868:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828707.009809:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828707.201927:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828708.033822:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828708.225899:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828709.057793:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828709.249826:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828710.081817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828710.273963:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828710.593913:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828710.593923:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828710.593928:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828710.593937:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5347 00000100:00080000:0.0:1787828710.593949:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828710.593956:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828710.593994:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790543104/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828710.594026:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828710.594029:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828710.594031:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828710.594034:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5347 00000100:00080000:0.0:1787828710.594037:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828710.594041:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828710.594050:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828710.594061:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790543232/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828710.594076:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828710.594082:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828710.594084:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828710.594088:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5347 00000100:00080000:0.0:1787828710.594091:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828710.594094:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828710.594099:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828710.594108:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790543360/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828710.594119:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828710.594124:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790543104:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828710.594126:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828710.594132:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828710.594140:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5347 00000100:00080000:0.0:1787828710.594146:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828710.594152:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828710.594160:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828710.594178:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790543488/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828710.594192:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828710.594197:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1787828710.594199:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828710.594201:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828710.594209:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5347 00000100:00080000:0.0:1787828710.594215:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828710.594219:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828710.594225:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828710.594237:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790543616/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828710.594246:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828710.594249:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828710.594252:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828710.594257:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5347 00000100:00100000:2.0:1787828710.594258:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828710/real 1787828710] req@ffff9e6803e9a1c0 x1874668790543104/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828710.594260:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828710.594269:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828710.594277:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828710.594301:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790543744/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828710.594301:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790543104/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828710.594348:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828710.594351:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828710.594352:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828710.594358:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828710.594363:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787828710.594368:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5347 00000100:00080000:0.0:1787828710.594374:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828710.594375:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790543104:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828710.594381:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828710.594386:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828710.594392:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790543232:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828710.594398:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790543872/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828710.594409:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828710.594412:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828710.594414:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828710.594417:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5347 00000100:00080000:0.0:1787828710.594421:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828710.594424:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828710.594428:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828710.594440:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790544000/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828710.594448:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828710.594454:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828710.594455:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828710.594486:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828710.594492:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5347 00000100:00080000:0.0:1787828710.594498:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828710.594500:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828710/real 1787828710] req@ffff9e6803e98b40 x1874668790543232/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828710.594506:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828710.594519:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828710.594529:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790543232/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828710.594541:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790544128/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828710.594543:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828710.594576:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828710.594591:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790543232:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828710.594647:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790543360:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828710.594693:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828710.594751:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828710/real 1787828710] req@ffff9e6803e98780 x1874668790543360/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828710.594763:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790543360/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828710.594786:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828710.594790:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828710.594794:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790543360:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828710.594801:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790543488:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828710.594828:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828710.594858:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828710/real 1787828710] req@ffff9e6803e9bc00 x1874668790543488/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828710.594874:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790543488/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828710.594886:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828710.594889:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828710.594893:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790543488:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828710.594900:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790543616:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828710.594950:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828710.594969:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828710/real 1787828710] req@ffff9e6803e9b840 x1874668790543616/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828710.594989:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790543616/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828710.595002:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828710.595006:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828710.595012:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790543616:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828710.595020:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790543744:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828710.595067:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828710.595083:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828710/real 1787828710] req@ffff9e6803e9b480 x1874668790543744/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828710.595099:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790543744/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828710.595113:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828710.595117:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828710.595122:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790543744:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828710.595131:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790543872:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828710.595171:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828710.595186:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828710/real 1787828710] req@ffff9e6803e9a580 x1874668790543872/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828710.595198:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790543872/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828710.595215:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828710.595218:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828710.595221:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790543872:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828710.595227:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790544000:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828710.595245:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828710.595255:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828710/real 1787828710] req@ffff9e6803e99680 x1874668790544000/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828710.595266:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790544000/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828710.595278:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828710.595283:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828710.595289:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790544000:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828710.595299:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790544128:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828710.595332:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828710.595366:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828710/real 1787828710] req@ffff9e6803e992c0 x1874668790544128/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828710.595382:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790544128/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828710.595392:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828710.595396:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828710.595401:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790544128:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828711.105808:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828711.297928:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828712.129881:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828712.321915:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828713.153818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828713.345945:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828714.177794:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828714.369878:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828715.201770:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828715.393925:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828715.713820:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828715.713829:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828715.713833:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828715.713843:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5352 00000100:00080000:0.0:1787828715.713853:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828715.713859:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828715.713891:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790544256/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828715.713920:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828715.713922:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828715.713923:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828715.713926:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5352 00000100:00080000:0.0:1787828715.713928:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828715.713931:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828715.713938:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828715.713947:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790544384/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828715.713954:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828715.713956:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828715.713958:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828715.713960:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5352 00000100:00080000:0.0:1787828715.713962:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828715.713965:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828715.713967:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828715.713975:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790544512/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828715.713982:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790544256:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828715.713987:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828715.713998:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828715.714000:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828715.714004:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5352 00000100:00080000:0.0:1787828715.714007:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828715.714009:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828715.714013:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828715.714021:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790544640/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828715.714034:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828715.714037:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828715.714040:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828715.714043:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5352 00000100:00080000:0.0:1787828715.714049:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828715.714052:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828715.714055:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828715.714065:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790544768/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828715.714075:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828715.714082:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828715.714089:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828715.714093:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828715.714099:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5352 00000100:00080000:0.0:1787828715.714108:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828715.714114:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828715.714121:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828715.714132:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828715/real 1787828715] req@ffff9e6803e9ad00 x1874668790544256/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828715.714142:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790544896/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828715.714155:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790544256/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828715.714203:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828715.714204:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828715.714218:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787828715.714220:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828715.714223:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790544256:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828715.714226:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828715.714236:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790544384:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828715.714237:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5352 00000100:00080000:0.0:1787828715.714249:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828715.714258:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787828715.714264:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787828715.714266:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828715.714276:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828715/real 1787828715] req@ffff9e6803e9a940 x1874668790544384/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828715.714287:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790545024/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828715.714288:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790544384/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828715.714354:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828715.714355:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828715.714364:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828715.714368:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787828715.714370:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828715.714373:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790544384:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828715.714382:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5352 00000100:00080000:0.0:1787828715.714388:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828715.714395:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828715.714401:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828715.714410:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790544512:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828715.714419:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790545152/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828715.714433:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828715.714434:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828715.714443:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828715.714447:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828715.714451:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828715/real 1787828715] req@ffff9e6803e99e00 x1874668790544512/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828715.714458:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5352 00000100:00080000:0.0:1787828715.714469:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828715.714479:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828715.714485:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790544512/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828715.714488:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828715.714508:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828715.714513:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790545280/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828715.714517:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828715.714527:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790544512:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828715.714567:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790544640:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828715.714593:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828715.714603:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828715/real 1787828715] req@ffff9e6803e98f00 x1874668790544640/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828715.714612:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790544640/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828715.714619:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828715.714621:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828715.714624:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790544640:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828715.714628:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790544768:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828715.714659:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828715.714667:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828715/real 1787828715] req@ffff9e6803e992c0 x1874668790544768/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828715.714676:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790544768/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828715.714683:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828715.714685:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828715.714703:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790544768:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828715.714708:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790544896:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828715.714769:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828715.714783:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828715/real 1787828715] req@ffff9e6803e99680 x1874668790544896/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828715.714795:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790544896/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828715.714806:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828715.714810:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828715.714814:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790544896:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828715.714821:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790545024:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828715.714843:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828715.714856:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828715/real 1787828715] req@ffff9e6803e9a580 x1874668790545024/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828715.714883:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790545024/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828715.714890:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828715.714892:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828715.714895:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790545024:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828715.714910:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790545152:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828715.714926:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828715.714933:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828715/real 1787828715] req@ffff9e6803e9b480 x1874668790545152/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828715.714941:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790545152/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828715.714947:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828715.714949:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828715.714951:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790545152:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828715.714955:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790545280:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828715.714965:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828715.714971:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828715/real 1787828715] req@ffff9e6803e9b840 x1874668790545280/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828715.714978:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790545280/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828715.714984:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828715.714986:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828715.714988:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790545280:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828716.225806:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828716.417912:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828717.249760:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828717.441858:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828718.273811:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828718.465867:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828719.297799:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828719.489856:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828720.321762:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828720.513907:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828720.833822:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828720.833830:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828720.833834:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828720.833843:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5357 00000100:00080000:0.0:1787828720.833853:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828720.833859:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828720.833893:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790545408/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828720.833919:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828720.833922:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828720.833924:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828720.833927:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5357 00000100:00080000:0.0:1787828720.833930:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828720.833932:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828720.833940:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828720.833950:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790545536/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828720.833969:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828720.833975:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828720.833976:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828720.833979:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5357 00000100:00080000:0.0:1787828720.833981:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828720.833986:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828720.833988:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790545408:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1787828720.833990:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828720.834001:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790545664/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828720.834009:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828720.834012:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828720.834013:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828720.834016:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5357 00000100:00080000:0.0:1787828720.834029:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828720.834032:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828720.834035:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828720.834044:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790545792/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828720.834051:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828720.834053:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828720.834054:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828720.834057:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5357 00000100:00080000:0.0:1787828720.834060:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828720.834065:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828720.834070:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828720.834075:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828720.834085:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790545920/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828720.834092:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828720.834095:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828720.834097:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828720.834101:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5357 00000100:00080000:0.0:1787828720.834104:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828720.834110:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828720.834110:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828720/real 1787828720] req@ffff9e6803e9bc00 x1874668790545408/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828720.834118:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828720.834132:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790546048/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828720.834136:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790545408/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828720.834156:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828720.834160:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828720.834160:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828720.834162:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828720.834167:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787828720.834170:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5357 00000100:00080000:0.0:1787828720.834174:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828720.834175:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790545408:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828720.834182:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828720.834187:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790545536:192.168.203.109@tcp:38:kworker.0 00000008:00100000:0.0:1787828720.834188:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828720.834208:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790546176/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828720.834217:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828720.834228:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828720.834231:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828720/real 1787828720] req@ffff9e6803e98780 x1874668790545536/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828720.834235:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828720.834241:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828720.834248:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790545536/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828720.834252:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5357 00000100:00080000:0.0:1787828720.834262:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828720.834264:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828720.834270:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787828720.834274:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828720.834276:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790545536:192.168.203.109@tcp:38:kworker.0 00000008:00100000:0.0:1787828720.834283:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828720.834299:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790546304/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828720.834302:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790545664:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828720.834319:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828720.834323:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828720.834325:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828720.834328:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828720.834333:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5357 00000100:00080000:0.0:1787828720.834338:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828720.834342:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828720/real 1787828720] req@ffff9e6803e98b40 x1874668790545664/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828720.834345:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828720.834353:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828720.834360:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790545664/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828720.834367:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790546432/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828720.834380:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828720.834388:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828720.834395:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790545664:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828720.834422:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790545792:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828720.834444:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828720.834453:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828720/real 1787828720] req@ffff9e6803e9a1c0 x1874668790545792/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828720.834463:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790545792/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828720.834470:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828720.834473:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828720.834475:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790545792:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828720.834480:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790545920:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828720.834492:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828720.834500:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828720/real 1787828720] req@ffff9e6803e9b840 x1874668790545920/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828720.834510:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790545920/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828720.834516:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828720.834519:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828720.834534:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790545920:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828720.834540:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790546048:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828720.834560:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828720.834584:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828720/real 1787828720] req@ffff9e6803e9b480 x1874668790546048/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828720.834592:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790546048/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828720.834612:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828720.834615:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828720.834617:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790546048:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828720.834621:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790546176:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828720.834638:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828720.834646:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828720/real 1787828720] req@ffff9e6803e9a580 x1874668790546176/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828720.834654:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790546176/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828720.834659:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828720.834661:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828720.834663:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790546176:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828720.834694:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790546304:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828720.834706:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828720.834742:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828720/real 1787828720] req@ffff9e6803e99680 x1874668790546304/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828720.834761:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790546304/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828720.834767:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828720.834769:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828720.834771:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790546304:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828720.834775:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790546432:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828720.834788:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828720.834797:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828720/real 1787828720] req@ffff9e6803e992c0 x1874668790546432/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828720.834811:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790546432/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828720.834827:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828720.834831:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828720.834835:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790546432:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828721.345821:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828721.537849:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828722.369770:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828722.561870:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828723.393864:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828723.586145:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828724.417797:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828724.609839:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828725.441802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828725.633867:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828725.953815:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828725.953823:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828725.953827:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828725.953836:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5362 00000100:00080000:3.0:1787828725.953858:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828725.953864:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828725.953897:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790546560/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828725.953920:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828725.953923:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828725.953925:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828725.953928:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5362 00000100:00080000:3.0:1787828725.953930:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828725.953933:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828725.953940:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828725.953950:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790546688/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828725.953959:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828725.953962:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828725.953964:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828725.953967:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5362 00000100:00080000:3.0:1787828725.953969:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828725.953972:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828725.953976:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828725.953982:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790546560:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828725.953985:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790546816/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828725.954000:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828725.954002:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828725.954003:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828725.954006:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5362 00000100:00080000:3.0:1787828725.954008:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828725.954011:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828725.954015:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828725.954022:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790546944/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828725.954029:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828725.954031:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828725.954032:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828725.954035:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5362 00000100:00080000:3.0:1787828725.954049:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828725.954052:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828725.954055:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828725.954067:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828725.954067:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790547072/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828725.954078:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828725.954081:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828725.954083:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828725.954086:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5362 00000100:00080000:3.0:1787828725.954089:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828725.954092:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828725.954096:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828725.954104:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790547200/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828725.954112:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828725.954117:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828725/real 1787828725] req@ffff9e680a2d6d00 x1874668790546560/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828725.954117:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828725.954121:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828725.954129:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5362 00000100:00080000:3.0:1787828725.954135:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828725.954142:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828725.954146:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828725.954150:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790546560/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828725.954157:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790547328/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828725.954178:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828725.954178:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828725.954183:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828725.954195:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828725.954199:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828725.954202:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5362 00000100:00080000:3.0:1787828725.954208:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828725.954213:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790546560:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828725.954213:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828725.954220:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828725.954234:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790546688:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828725.954235:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790547456/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828725.954260:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828725.954263:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828725.954265:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828725.954268:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5362 00000100:00080000:3.0:1787828725.954272:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828725.954275:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828725.954279:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828725.954290:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828725.954290:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790547584/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828725.954312:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828725/real 1787828725] req@ffff9e680a2d5e00 x1874668790546688/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828725.954335:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790546688/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828725.954343:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828725.954345:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828725.954348:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790546688:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828725.954374:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790546816:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828725.954389:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828725.954397:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828725/real 1787828725] req@ffff9e680a2d7480 x1874668790546816/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828725.954406:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790546816/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828725.954412:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828725.954415:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828725.954417:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790546816:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828725.954422:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790546944:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828725.954433:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828725.954440:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828725/real 1787828725] req@ffff9e680a2d4b40 x1874668790546944/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828725.954448:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790546944/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828725.954454:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828725.954456:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828725.954458:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790546944:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828725.954461:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790547072:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828725.954472:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828725.954479:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828725/real 1787828725] req@ffff9e680a2d61c0 x1874668790547072/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828725.954486:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790547072/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828725.954492:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828725.954494:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828725.954496:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790547072:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828725.954499:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790547200:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828725.954510:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828725.954516:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828725/real 1787828725] req@ffff9e680a2d4f00 x1874668790547200/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828725.954524:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790547200/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828725.954531:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828725.954533:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828725.954535:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790547200:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828725.954539:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790547328:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828725.954550:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828725.954556:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828725/real 1787828725] req@ffff9e680a2d7c00 x1874668790547328/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828725.954584:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790547328/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828725.954590:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828725.954592:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828725.954595:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790547328:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828725.954599:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790547456:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828725.954615:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828725.954624:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828725/real 1787828725] req@ffff9e680a2d70c0 x1874668790547456/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828725.954632:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790547456/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828725.954643:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828725.954645:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828725.954648:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790547456:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828725.954652:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790547584:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828725.954664:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828725.954670:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828725/real 1787828725] req@ffff9e680a2d52c0 x1874668790547584/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828725.954678:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790547584/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828750 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828725.954683:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828725.954685:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828725.954687:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790547584:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828726.465940:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828726.657839:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828727.489840:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828727.681969:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828728.513801:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828728.705845:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828729.537815:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828729.729901:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828730.561848:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828730.753884:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828731.073834:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828731.073849:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828731.073855:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828731.073868:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5367 00000100:00080000:1.0:1787828731.073883:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828731.073893:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828731.073943:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790547712/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828731.073982:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828731.073988:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828731.073990:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828731.073996:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5367 00000100:00080000:1.0:1787828731.074001:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828731.074006:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828731.074021:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828731.074032:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790547712:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828731.074041:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790547840/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828731.074054:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828731.074059:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828731.074062:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828731.074069:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5367 00000100:00080000:1.0:1787828731.074075:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828731.074080:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828731.074085:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828731.074096:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790547968/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828731.074107:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828731.074111:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828731.074113:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828731.074117:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5367 00000100:00080000:1.0:1787828731.074123:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828731.074128:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828731.074133:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828731.074140:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828731.074157:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790548096/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828731.074169:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828731.074173:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828731.074176:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828731.074185:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5367 00000100:00080000:1.0:1787828731.074192:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828731.074194:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828731/real 1787828731] req@ffff9e6831aa8000 x1874668790547712/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828731.074203:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828731.074212:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828731.074243:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790548224/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828731.074254:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790547712/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828731.074259:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828731.074268:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828731.074273:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828731.074278:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828731.074281:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5367 00000100:00080000:0.0:1787828731.074289:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828731.074289:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828731.074300:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828731.074302:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790547712:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787828731.074310:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828731.074329:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790548352/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828731.074350:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828731.074353:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790547840:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828731.074355:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828731.074360:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828731.074368:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5367 00000100:00080000:1.0:1787828731.074372:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828731.074377:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828731.074383:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828731.074393:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828731.074398:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790548480/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828731.074409:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828731.074415:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787828731.074417:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828731/real 1787828731] req@ffff9e6831aaa1c0 x1874668790547840/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828731.074418:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828731.074426:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5367 00000100:00080000:1.0:1787828731.074434:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828731.074443:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828731.074448:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828731.074450:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790547840/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828731.074464:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790548608/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828731.074470:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828731.074478:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828731.074482:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828731.074486:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787828731.074491:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790547840:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828731.074493:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828731.074503:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5367 00000100:00100000:0.0:1787828731.074509:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790547968:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828731.074510:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828731.074521:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828731.074542:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828731.074558:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828731.074560:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790548736/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828731.074578:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828731/real 1787828731] req@ffff9e6831aa8780 x1874668790547968/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828731.074597:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790547968/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828731.074613:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828731.074617:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828731.074621:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790547968:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828731.074628:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790548096:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828731.074651:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828731.074664:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828731/real 1787828731] req@ffff9e6831aa8b40 x1874668790548096/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828731.074689:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790548096/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828731.074749:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828731.074754:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828731.074758:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790548096:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828731.074765:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790548224:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828731.074790:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828731.074803:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828731/real 1787828731] req@ffff9e6831aaad00 x1874668790548224/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828731.074817:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790548224/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828731.074825:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828731.074829:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828731.074833:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790548224:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828731.074854:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790548352:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828731.074879:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828731.074891:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828731/real 1787828731] req@ffff9e6831aab480 x1874668790548352/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828731.074904:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790548352/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828731.074915:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828731.074919:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828731.074923:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790548352:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828731.074930:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790548480:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828731.074952:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828731.074965:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828731/real 1787828731] req@ffff9e6831aabc00 x1874668790548480/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828731.074978:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790548480/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828731.074986:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828731.074990:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828731.074994:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790548480:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828731.075001:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790548608:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828731.075023:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828731.075038:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828731/real 1787828731] req@ffff9e6831aa9a40 x1874668790548608/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828731.075051:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790548608/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828731.075061:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828731.075064:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828731.075068:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790548608:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828731.075075:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790548736:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828731.075096:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828731.075108:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828731/real 1787828731] req@ffff9e6831aaa940 x1874668790548736/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828731.075121:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790548736/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828731.075130:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828731.075134:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828731.075137:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790548736:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828731.585845:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828731.777957:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828732.609800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828732.801892:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828733.633784:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828733.825915:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828734.657796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828734.849955:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828735.681792:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828735.873843:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828736.193776:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828736.193783:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828736.193788:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828736.193796:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5372 00000100:00080000:1.0:1787828736.193807:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828736.193813:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828736.193843:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790548864/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828736.193864:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828736.193867:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828736.193868:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828736.193871:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5372 00000100:00080000:1.0:1787828736.193873:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828736.193876:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828736.193883:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828736.193893:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790548992/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828736.193901:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828736.193903:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828736.193905:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828736.193910:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790548864:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828736.193910:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5372 00000100:00080000:1.0:1787828736.193914:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828736.193919:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828736.193923:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828736.193931:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790549120/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828736.193937:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828736.193940:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828736.193941:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828736.193944:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5372 00000100:00080000:1.0:1787828736.193946:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828736.193948:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828736.193951:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828736.193958:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790549248/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828736.193964:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828736.193966:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828736.193967:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828736.193969:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5372 00000100:00080000:1.0:1787828736.193972:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828736.193975:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828736.193978:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828736.193982:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828736.193995:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790549376/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828736.194003:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828736.194005:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828736.194006:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828736.194009:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5372 00000100:00080000:1.0:1787828736.194012:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828736.194015:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828736.194018:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828736.194031:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790549504/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828736.194032:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828736/real 1787828736] req@ffff9e6831aa9e00 x1874668790548864/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828736.194045:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828736.194051:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828736.194054:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828736.194059:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5372 00000100:00080000:1.0:1787828736.194062:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828736.194065:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828736.194067:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828736.194071:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790548864/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828736.194076:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790549632/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828736.194096:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828736.194099:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828736.194102:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828736.194105:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828736.194105:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828736.194114:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5372 00000100:00100000:0.0:1787828736.194117:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790548864:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828736.194117:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828736.194124:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828736.194128:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828736.194134:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790549760/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828736.194139:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828736.194142:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828736.194143:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828736.194147:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5372 00000100:00100000:0.0:1787828736.194149:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790548992:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828736.194150:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828736.194155:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828736.194159:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828736.194167:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790549888/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828736.194184:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828736.194199:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828736/real 1787828736] req@ffff9e6831aa9680 x1874668790548992/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828736.194213:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790548992/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828736.194223:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828736.194226:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828736.194229:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790548992:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828736.194235:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790549120:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828736.194253:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828736.194264:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828736/real 1787828736] req@ffff9e6831aaa580 x1874668790549120/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828736.194276:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790549120/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828736.194289:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828736.194292:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828736.194295:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790549120:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828736.194301:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790549248:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828736.194319:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828736.194328:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828736/real 1787828736] req@ffff9e6831aa8f00 x1874668790549248/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828736.194341:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790549248/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828736.194350:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828736.194353:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828736.194356:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790549248:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828736.194363:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790549376:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828736.194383:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828736.194394:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828736/real 1787828736] req@ffff9e6831aaa940 x1874668790549376/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828736.194402:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790549376/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828736.194408:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828736.194410:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828736.194413:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790549376:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828736.194417:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790549504:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828736.194430:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828736.194436:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828736/real 1787828736] req@ffff9e6831aa9a40 x1874668790549504/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828736.194443:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790549504/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828736.194449:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828736.194451:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828736.194453:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790549504:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828736.194456:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790549632:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828736.194466:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828736.194471:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828736/real 1787828736] req@ffff9e6831aabc00 x1874668790549632/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828736.194478:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790549632/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828736.194483:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828736.194485:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828736.194486:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790549632:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828736.194490:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790549760:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828736.194520:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828736.194528:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828736/real 1787828736] req@ffff9e6831aab480 x1874668790549760/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828736.194536:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790549760/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828736.194541:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828736.194543:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828736.194545:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790549760:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828736.194559:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790549888:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828736.194569:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828736.194575:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828736/real 1787828736] req@ffff9e6831aaad00 x1874668790549888/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828736.194582:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790549888/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828736.194587:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828736.194588:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828736.194590:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790549888:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828736.705797:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828736.897859:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828737.729801:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828737.921890:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828738.753829:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828738.945943:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828739.777829:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828739.970056:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828740.801895:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828740.993883:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828741.313848:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828741.313861:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828741.313868:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828741.313883:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5378 00000100:00080000:1.0:1787828741.313900:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828741.313913:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828741.313965:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790550016/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828741.313990:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828741.313994:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828741.313996:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828741.313999:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5378 00000100:00080000:1.0:1787828741.314002:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828741.314005:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828741.314015:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828741.314027:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790550144/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828741.314037:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828741.314040:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828741.314042:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828741.314047:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5378 00000100:00080000:1.0:1787828741.314049:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828741.314054:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828741.314057:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790550016:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787828741.314059:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828741.314078:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790550272/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828741.314097:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828741.314102:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828741.314104:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828741.314110:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5378 00000100:00080000:1.0:1787828741.314125:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828741.314132:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787828741.314135:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1787828741.314147:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828741.314162:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790550400/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828741.314172:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828741.314177:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787828741.314179:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828741/real 1787828741] req@ffff9e6831aa8b40 x1874668790550016/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828741.314181:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828741.314201:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5378 00000100:00100000:0.0:1787828741.314207:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790550016/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828741.314208:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828741.314222:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787828741.314225:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828741.314234:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:1.0:1787828741.314234:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828741.314248:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790550016:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828741.314259:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790550528/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828741.314269:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828741.314272:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787828741.314277:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790550144:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828741.314278:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828741.314294:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5378 00000100:00080000:1.0:1787828741.314301:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828741.314305:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828741.314311:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828741.314318:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828741.314326:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790550656/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828741.314339:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828741.314345:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787828741.314346:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828741/real 1787828741] req@ffff9e6831aa8780 x1874668790550144/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828741.314347:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828741.314368:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5378 00000100:00080000:1.0:1787828741.314373:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828741.314381:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828741.314390:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828741.314408:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790550784/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828741.314420:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790550144/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828741.314422:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828741.314433:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828741.314438:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828741.314471:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5378 00000100:00080000:1.0:1787828741.314478:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828741.314484:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828741.314484:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787828741.314501:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:1.0:1787828741.314501:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828741.314514:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790550144:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828741.314521:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790550912/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828741.314528:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790550272:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828741.314558:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828741.314561:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828741.314564:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828741.314569:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5378 00000100:00080000:1.0:1787828741.314575:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828741.314577:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828741.314582:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828741.314587:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828741.314596:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828741/real 1787828741] req@ffff9e6831aaa1c0 x1874668790550272/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828741.314602:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790551040/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828741.314661:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790550272/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828741.314677:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828741.314682:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828741.314685:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790550272:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828741.314691:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790550400:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828741.314764:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828741.314773:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828741/real 1787828741] req@ffff9e6831aa8000 x1874668790550400/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828741.314781:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790550400/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828741.314787:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828741.314790:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828741.314792:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790550400:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828741.314796:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790550528:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828741.314808:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828741.314814:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828741/real 1787828741] req@ffff9e6831aaad00 x1874668790550528/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828741.314822:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790550528/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828741.314827:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828741.314828:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828741.314831:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790550528:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828741.314846:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790550656:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828741.314857:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828741.314863:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828741/real 1787828741] req@ffff9e6831aab480 x1874668790550656/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828741.314871:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790550656/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828741.314878:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828741.314879:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828741.314881:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790550656:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828741.314885:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790550784:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828741.314895:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828741.314901:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828741/real 1787828741] req@ffff9e6831aabc00 x1874668790550784/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828741.314908:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790550784/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828741.314914:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828741.314915:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828741.314917:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790550784:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828741.314921:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790550912:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828741.314930:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828741.314937:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828741/real 1787828741] req@ffff9e6831aa9a40 x1874668790550912/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828741.314944:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790550912/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828741.314949:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828741.314951:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828741.314953:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790550912:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828741.314957:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790551040:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828741.314966:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828741.314972:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828741/real 1787828741] req@ffff9e6831aaa940 x1874668790551040/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828741.314978:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790551040/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828741.314983:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828741.314985:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828741.314987:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790551040:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828741.825825:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828742.017938:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828742.849829:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828743.042031:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828743.873884:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828744.065991:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828744.897850:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828745.089888:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828745.921874:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828746.113982:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828746.433842:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828746.433852:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828746.433856:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828746.433884:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5383 00000100:00080000:2.0:1787828746.433907:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828746.433915:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828746.433964:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790551168/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828746.434007:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828746.434015:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828746.434018:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828746.434025:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5383 00000100:00080000:2.0:1787828746.434032:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828746.434040:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828746.434069:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828746.434088:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790551296/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828746.434101:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828746.434104:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828746.434106:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828746.434113:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790551168:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828746.434114:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5383 00000100:00080000:2.0:1787828746.434122:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828746.434128:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828746.434133:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828746.434147:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790551424/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828746.434156:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828746.434160:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828746.434162:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828746.434165:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5383 00000100:00080000:2.0:1787828746.434168:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828746.434172:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828746.434177:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828746.434187:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790551552/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828746.434196:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787828746.434204:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828746.434204:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828746.434210:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828746.434215:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5383 00000100:00080000:2.0:1787828746.434218:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828746.434224:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828746.434230:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828746.434248:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790551680/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828746.434262:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828746.434269:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828746/real 1787828746] req@ffff9e6804423840 x1874668790551168/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828746.434270:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828746.434276:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828746.434291:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5383 00000100:00080000:2.0:1787828746.434302:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828746.434309:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828746.434316:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790551168/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828746.434316:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787828746.434332:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787828746.434339:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790551808/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828746.434340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828746.434355:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790551168:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828746.434373:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828746.434387:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790551296:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828746.434392:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828746.434401:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828746.434406:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5383 00000100:00080000:2.0:1787828746.434411:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828746.434415:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828746.434439:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828746.434445:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828746.434457:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790551936/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828746.434496:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828746/real 1787828746] req@ffff9e68044221c0 x1874668790551296/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828746.434505:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828746.434517:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828746.434520:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828746.434527:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790551296/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828746.434533:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5383 00000100:00080000:2.0:1787828746.434543:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828746.434568:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828746.434569:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787828746.434579:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:2.0:1787828746.434580:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828746.434585:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790551296:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828746.434597:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790552064/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828746.434598:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790551424:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828746.434627:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828746.434631:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828746.434632:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828746.434638:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5383 00000800:00000100:0.0:1787828746.434643:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828746.434646:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828746.434653:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828746.434657:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828746.434659:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828746/real 1787828746] req@ffff9e6804423480 x1874668790551424/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828746.434672:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790552192/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828746.434676:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790551424/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828746.434710:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828746.434714:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828746.434717:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790551424:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828746.434757:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790551552:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828746.434778:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828746.434788:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828746/real 1787828746] req@ffff9e68044212c0 x1874668790551552/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828746.434798:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790551552/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828746.434806:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828746.434808:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828746.434811:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790551552:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828746.434816:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790551680:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828746.434849:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828746.434860:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828746/real 1787828746] req@ffff9e68044230c0 x1874668790551680/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828746.434871:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790551680/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828746.434880:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828746.434882:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828746.434887:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790551680:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828746.434893:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790551808:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828746.434910:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828746.434919:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828746/real 1787828746] req@ffff9e6804420b40 x1874668790551808/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828746.434950:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790551808/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828746.434961:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828746.434965:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828746.434968:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790551808:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828746.434992:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790551936:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828746.435015:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828746.435027:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828746/real 1787828746] req@ffff9e6804422580 x1874668790551936/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828746.435041:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790551936/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828746.435050:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828746.435069:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828746.435073:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790551936:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828746.435078:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790552064:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828746.435093:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828746.435101:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828746/real 1787828746] req@ffff9e6804421680 x1874668790552064/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828746.435111:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790552064/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828746.435118:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828746.435120:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828746.435123:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790552064:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828746.435127:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790552192:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828746.435140:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828746.435153:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828746/real 1787828746] req@ffff9e6804420780 x1874668790552192/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828746.435168:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790552192/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828746.435180:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828746.435185:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828746.435190:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790552192:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828746.945833:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828747.137859:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828747.969793:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828748.161824:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828748.993785:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828749.185952:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828750.017882:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828750.209899:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828751.041759:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828751.233912:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828751.553780:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828751.553792:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828751.553797:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828751.553808:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5388 00000100:00080000:2.0:1787828751.553818:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828751.553826:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828751.553863:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790552320/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828751.553887:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828751.553890:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828751.553892:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828751.553894:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5388 00000100:00080000:2.0:1787828751.553898:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828751.553902:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828751.553910:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828751.553921:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790552320:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787828751.553922:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790552448/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828751.553936:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828751.553938:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828751.553939:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828751.553942:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5388 00000100:00080000:2.0:1787828751.553943:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828751.553947:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828751.553950:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828751.553957:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790552576/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828751.553964:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828751.553968:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828751.553970:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828751.553973:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5388 00000100:00080000:2.0:1787828751.553978:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828751.553982:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787828751.553987:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:2.0:1787828751.553988:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828751.554006:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790552704/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828751.554017:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828751.554021:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787828751.554025:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828751/real 1787828751] req@ffff9e6804422d00 x1874668790552320/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828751.554025:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828751.554033:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5388 00000100:00080000:2.0:1787828751.554039:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828751.554045:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828751.554049:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790552320/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828751.554051:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787828751.554064:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787828751.554068:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790552832/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828751.554071:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828751.554079:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790552320:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828751.554083:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828751.554087:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828751.554088:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828751.554092:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5388 00000100:00080000:2.0:1787828751.554094:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828751.554097:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828751.554099:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828751.554104:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790552448:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828751.554109:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790552960/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828751.554118:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828751.554120:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828751.554122:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828751.554125:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5388 00000800:00000100:0.0:1787828751.554129:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828751.554131:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828751.554135:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828751.554139:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828751.554143:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828751/real 1787828751] req@ffff9e6804422940 x1874668790552448/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828751.554149:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790553088/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828751.554163:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790552448/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828751.554164:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828751.554172:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828751.554175:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828751.554182:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5388 00000100:00080000:0.0:1787828751.554183:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828751.554185:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828751.554189:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787828751.554193:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828751.554195:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790552448:192.168.203.109@tcp:38:kworker.0 00000008:00100000:2.0:1787828751.554198:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828751.554203:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790552576:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828751.554210:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790553216/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828751.554218:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787828751.554222:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828751.554223:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828751.554226:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828751.554231:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5388 00000100:00080000:2.0:1787828751.554234:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828751.554239:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828751.554240:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828751/real 1787828751] req@ffff9e6804421e00 x1874668790552576/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828751.554243:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828751.554254:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790553344/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828751.554263:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790552576/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828751.554276:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828751.554280:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828751.554283:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790552576:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828751.554289:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790552704:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828751.554305:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828751.554314:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828751/real 1787828751] req@ffff9e6804420f00 x1874668790552704/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828751.554321:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790552704/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828751.554327:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828751.554329:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828751.554331:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790552704:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828751.554335:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790552832:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828751.554345:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828751.554351:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828751/real 1787828751] req@ffff9e6804420780 x1874668790552832/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828751.554358:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790552832/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828751.554363:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828751.554364:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828751.554366:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790552832:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828751.554377:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790552960:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828751.554404:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828751.554413:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828751/real 1787828751] req@ffff9e6804421680 x1874668790552960/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828751.554421:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790552960/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828751.554428:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828751.554430:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828751.554432:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790552960:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828751.554436:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790553088:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828751.554447:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828751.554453:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828751/real 1787828751] req@ffff9e6804422580 x1874668790553088/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828751.554461:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790553088/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828751.554466:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828751.554467:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828751.554469:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790553088:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828751.554472:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790553216:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828751.554482:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828751.554487:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828751/real 1787828751] req@ffff9e6804420b40 x1874668790553216/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828751.554495:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790553216/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828751.554504:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828751.554507:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828751.554509:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790553216:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828751.554512:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790553344:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828751.554521:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828751.554526:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828751/real 1787828751] req@ffff9e68044230c0 x1874668790553344/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828751.554533:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790553344/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828751.554538:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828751.554540:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828751.554542:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790553344:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828752.065780:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828752.257947:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828753.089836:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828753.281867:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828754.113835:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828754.306019:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828755.137843:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828755.329961:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828756.161841:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828756.354016:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828756.673813:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828756.673823:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828756.673827:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828756.673836:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5393 00000100:00080000:2.0:1787828756.673846:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828756.673852:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828756.673884:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790553472/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828756.673907:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828756.673910:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828756.673911:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828756.673914:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5393 00000100:00080000:2.0:1787828756.673917:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828756.673921:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828756.673929:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828756.673937:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790553600/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828756.673946:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828756.673948:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828756.673949:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828756.673955:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5393 00000100:00080000:2.0:1787828756.673958:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828756.673965:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828756.673967:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790553472:192.168.203.109@tcp:250:kworker.0 00000008:00100000:2.0:1787828756.673970:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828756.673981:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790553728/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828756.673988:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828756.673990:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828756.673991:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828756.673995:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5393 00000100:00080000:2.0:1787828756.673997:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828756.673999:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828756.674003:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828756.674011:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790553856/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828756.674022:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828756.674024:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828756.674025:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828756.674027:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5393 00000100:00080000:2.0:1787828756.674033:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828756.674035:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828756.674038:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828756.674046:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790553984/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828756.674052:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828756.674055:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828756.674056:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828756.674065:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5393 00000800:00000100:0.0:1787828756.674068:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828756.674071:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828756.674075:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828756.674080:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828756.674092:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790554112/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828756.674098:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828756.674100:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828756.674102:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828756.674107:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5393 00000100:00080000:2.0:1787828756.674111:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828756.674116:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828756.674123:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828756/real 1787828756] req@ffff9e68044212c0 x1874668790553472/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828756.674134:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828756.674152:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790553472/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828756.674152:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790554240/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828756.674219:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828756.674220:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828756.674231:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828756.674233:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787828756.674235:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828756.674240:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790553472:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828756.674242:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5393 00000100:00080000:2.0:1787828756.674252:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828756.674258:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828756.674262:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828756.674270:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790554368/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828756.674280:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828756.674281:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790553600:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828756.674290:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828756.674296:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828756.674303:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5393 00000100:00080000:2.0:1787828756.674307:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828756.674312:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828756.674317:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828756.674323:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828756.674327:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790554496/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828756.674338:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828756/real 1787828756] req@ffff9e6804423480 x1874668790553600/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828756.674377:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790553600/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828756.674389:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828756.674401:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828756.674405:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790553600:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828756.674414:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790553728:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828756.674440:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828756.674454:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828756/real 1787828756] req@ffff9e68044221c0 x1874668790553728/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828756.674467:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790553728/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828756.674477:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828756.674481:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828756.674484:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790553728:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828756.674492:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790553856:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828756.674515:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828756.674525:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828756/real 1787828756] req@ffff9e6804423840 x1874668790553856/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828756.674534:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790553856/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828756.674540:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828756.674544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828756.674547:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790553856:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828756.674554:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790553984:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828756.674576:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828756.674588:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828756/real 1787828756] req@ffff9e68044230c0 x1874668790553984/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828756.674602:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790553984/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828756.674612:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828756.674615:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828756.674619:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790553984:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828756.674626:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790554112:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828756.674644:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828756.674652:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828756/real 1787828756] req@ffff9e6804420b40 x1874668790554112/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828756.674660:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790554112/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828756.674672:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828756.674674:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828756.674676:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790554112:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828756.674680:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790554240:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828756.674690:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828756.674720:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828756/real 1787828756] req@ffff9e6804422580 x1874668790554240/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828756.674729:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790554240/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828756.674734:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828756.674736:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828756.674738:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790554240:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828756.674759:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790554368:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828756.674781:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828756.674794:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828756/real 1787828756] req@ffff9e6804421680 x1874668790554368/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828756.674807:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790554368/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828756.674814:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828756.674816:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828756.674819:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790554368:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828756.674824:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790554496:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828756.674840:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828756.674847:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828756/real 1787828756] req@ffff9e6804420780 x1874668790554496/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828756.674855:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790554496/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828756.674860:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828756.674862:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828756.674864:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790554496:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828757.185802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828757.377924:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828758.209792:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828758.401885:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828759.233762:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828759.425949:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828760.257841:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828760.449892:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828761.281767:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828761.473972:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828761.793831:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828761.793845:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828761.793850:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828761.793862:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5398 00000100:00080000:2.0:1787828761.793875:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828761.793885:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828761.793947:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790554624/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828761.793984:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828761.793989:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828761.793992:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828761.793998:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5398 00000100:00080000:2.0:1787828761.794003:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828761.794010:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828761.794024:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828761.794045:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790554624:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787828761.794052:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790554752/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828761.794090:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828761.794097:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828761.794102:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828761.794111:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5398 00000100:00080000:2.0:1787828761.794120:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828761.794129:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828761.794141:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828761.794159:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828761.794166:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790554880/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828761.794183:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828761.794188:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828761.794191:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828761.794199:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5398 00000100:00080000:2.0:1787828761.794205:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828761.794211:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828761.794219:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828761.794229:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828761/real 1787828761] req@ffff9e6804420f00 x1874668790554624/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828761.794235:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790555008/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828761.794303:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828761.794309:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828761.794311:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828761.794327:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790554624/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828761.794347:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828761.794348:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5398 00000100:00080000:2.0:1787828761.794365:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828761.794366:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787828761.794386:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828761.794388:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790554624:192.168.203.109@tcp:250:kworker.0 00000008:00100000:2.0:1787828761.794410:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828761.794433:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790555136/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828761.794439:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790554752:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828761.794474:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828761.794480:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828761.794484:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828761.794492:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5398 00000100:00080000:2.0:1787828761.794498:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828761.794505:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828761.794522:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828761.794530:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828761/real 1787828761] req@ffff9e6804421e00 x1874668790554752/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828761.794533:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828761.794560:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790554752/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828761.794569:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790555264/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828761.794582:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828761.794591:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828761.794600:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790554752:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828761.794622:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790554880:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828761.794640:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828761.794649:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:0.0:1787828761.794650:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828761.794660:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828761.794670:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828761/real 1787828761] req@ffff9e6804422940 x1874668790554880/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828761.794677:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5398 00000100:00100000:0.0:1787828761.794726:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790554880/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828761.794734:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828761.794738:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828761.794741:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790554880:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828761.794746:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828761.794748:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790555008:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787828761.794766:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828761.794779:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828761.794795:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828761.794811:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790555392/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828761.794819:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828761/real 1787828761] req@ffff9e6804422d00 x1874668790555008/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828761.794888:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828761.794898:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790555008/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828761.794902:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828761.794911:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828761.794921:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828761.794930:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5398 00000100:00080000:0.0:1787828761.794934:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828761.794946:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790555008:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787828761.794948:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828761.794960:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828761.794970:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790555136:192.168.203.109@tcp:8:kworker.0 00000008:00100000:2.0:1787828761.794971:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828761.795012:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790555520/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828761.795025:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828761.795043:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828761/real 1787828761] req@ffff9e6804420780 x1874668790555136/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828761.795049:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828761.795057:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790555136/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828761.795064:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828761.795077:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828761.795080:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828761.795101:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787828761.795102:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5398 00000100:00100000:0.0:1787828761.795110:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790555136:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787828761.795119:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828761.795120:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790555264:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828761.795141:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787828761.795150:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:2.0:1787828761.795150:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828761.795164:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828761/real 1787828761] req@ffff9e6804421680 x1874668790555264/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828761.795175:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790555648/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828761.795178:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790555264/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828761.795268:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828761.795278:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828761.795287:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790555264:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828761.795298:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790555392:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828761.795335:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828761.795345:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828761/real 1787828761] req@ffff9e6804422580 x1874668790555392/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828761.795353:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790555392/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828761.795359:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828761.795361:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828761.795363:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790555392:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828761.795374:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790555520:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828761.795387:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828761.795393:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828761/real 1787828761] req@ffff9e6804420b40 x1874668790555520/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828761.795404:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790555520/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828761.795411:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828761.795415:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828761.795417:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790555520:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828761.795427:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790555648:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828761.795439:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828761.795446:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828761/real 1787828761] req@ffff9e68044230c0 x1874668790555648/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828761.795453:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790555648/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828761.795459:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828761.795461:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828761.795462:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790555648:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828762.305782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828762.497983:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828763.329853:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828763.521951:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828764.353812:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828764.545939:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828765.377860:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828765.569933:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828766.401834:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828766.593983:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828766.913850:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828766.913864:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828766.913868:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828766.913878:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5403 00000100:00080000:2.0:1787828766.913890:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828766.913898:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828766.913942:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790555776/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828766.913986:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828766.913990:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828766.913992:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828766.913997:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5403 00000100:00080000:2.0:1787828766.914000:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828766.914005:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828766.914015:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828766.914028:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790555904/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828766.914037:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828766.914040:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828766.914041:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828766.914044:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5403 00000100:00080000:2.0:1787828766.914047:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828766.914049:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828766.914054:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828766.914066:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790556032/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828766.914073:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828766.914076:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828766.914078:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828766.914081:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5403 00000100:00080000:2.0:1787828766.914085:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828766.914088:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828766.914092:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828766.914112:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790556160/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828766.914131:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828766.914136:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790555776:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828766.914137:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828766.914142:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828766.914149:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5403 00000100:00080000:2.0:1787828766.914153:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828766.914157:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828766.914173:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828766.914187:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790556288/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828766.914197:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828766.914200:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828766.914202:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828766.914207:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5403 00000100:00080000:2.0:1787828766.914212:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828766.914216:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828766.914222:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828766.914234:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790556416/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828766.914244:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828766.914248:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828766.914250:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828766.914255:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790555904:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828766.914257:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5403 00000100:00080000:2.0:1787828766.914263:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828766.914267:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828766.914273:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828766.914304:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790556544/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828766.914324:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828766.914327:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790556032:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828766.914329:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828766.914335:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828766.914342:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5403 00000100:00080000:2.0:1787828766.914348:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828766.914352:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828766.914359:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828766.914371:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790556160:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828766.914373:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790556672/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828766.914391:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828766.914394:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828766.914396:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828766.914401:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5403 00000100:00100000:0.0:1787828766.914405:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790556288:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787828766.914406:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828766.914415:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828766.914421:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828766.914435:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790556800/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828766.914440:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790556416:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828766.914474:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790556544:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828766.914497:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790556672:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828766.914514:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790556800:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828767.425801:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828767.617898:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828768.449782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828768.641909:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828769.473878:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828769.665906:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787828769.985981:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:1.0:1787828769.986022:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787828769.986035:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787828769.986081:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:1.0:1787828769.986102:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787828769.986113:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1787828769.986121:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828766/real 1787828769] req@ffff9e6804420780 x1874668790556800/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787828769.986122:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787828769.986135:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787828769.986148:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1787828769.986164:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790556800/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787828769.986165:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00080000:0.0:1787828769.986184:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000800:00000100:1.0:1787828769.986186:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00080000:0.0:1787828769.986195:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828769.986202:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790556800:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828769.986211:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828766/real 1787828769] req@ffff9e68044212c0 x1874668790556160/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828769.986228:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790556160/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828769.986236:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828769.986240:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828769.986244:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790556160:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828769.986250:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828766/real 1787828769] req@ffff9e68044230c0 x1874668790556288/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828769.986279:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790556288/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828769.986288:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828769.986292:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828769.986295:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790556288:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828769.986302:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828766/real 1787828769] req@ffff9e6804420b40 x1874668790556416/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828769.986313:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790556416/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828769.986322:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828769.986326:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828769.986329:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790556416:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828769.986340:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828766/real 1787828769] req@ffff9e6804422580 x1874668790556544/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828769.986351:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790556544/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828769.986358:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828769.986361:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828769.986365:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790556544:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828769.986371:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828766/real 1787828769] req@ffff9e6804423840 x1874668790555776/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828769.986383:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790555776/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828769.986391:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828769.986394:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828769.986397:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790555776:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787828769.986403:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828766/real 1787828769] req@ffff9e68044221c0 x1874668790555904/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828769.986415:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790555904/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828769.986424:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828769.986427:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828769.986430:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790555904:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828769.986436:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828766/real 1787828769] req@ffff9e6804423480 x1874668790556032/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828769.986448:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790556032/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828769.986456:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828769.986460:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828769.986463:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790556032:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828769.986496:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828766/real 1787828769] req@ffff9e6804421680 x1874668790556672/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828769.986509:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790556672/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828791 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828769.986517:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828769.986521:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828769.986524:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790556672:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828770.497813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828770.689885:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828771.521844:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828771.713924:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828772.033844:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828772.033851:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828772.033855:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828772.033876:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5408 00000100:00080000:3.0:1787828772.033886:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828772.033892:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828772.033924:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790556928/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828772.033957:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828772.033959:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828772.033961:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828772.033964:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5408 00000100:00080000:3.0:1787828772.033966:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828772.033968:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828772.033976:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828772.033990:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790557056/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828772.034012:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828772.034015:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828772.034018:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828772.034023:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5408 00000100:00080000:3.0:1787828772.034027:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828772.034035:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790556928:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828772.034037:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828772.034053:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828772.034073:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790557184/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828772.034086:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828772.034090:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828772.034091:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828772.034095:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5408 00000100:00080000:3.0:1787828772.034101:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828772.034105:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828772.034110:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828772.034122:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790557312/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828772.034134:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787828772.034137:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828772.034151:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828772.034155:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828772.034165:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5408 00000100:00080000:3.0:1787828772.034171:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828772.034178:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828772.034186:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828772.034205:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828772/real 1787828772] req@ffff9e680a2d4000 x1874668790556928/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828772.034208:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790557440/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828772.034272:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828772.034285:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790556928/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828772.034285:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828772.034293:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828772.034308:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5408 00000100:00080000:0.0:1787828772.034322:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828772.034331:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828772.034336:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828772.034342:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790556928:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828772.034355:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828772.034368:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828772.034374:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790557056:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828772.034391:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790557568/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828772.034404:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828772.034408:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828772.034411:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828772.034420:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5408 00000800:00000100:0.0:1787828772.034423:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828772.034430:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828772.034452:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828772.034460:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828772/real 1787828772] req@ffff9e680a2d4780 x1874668790557056/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828772.034464:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828772.034490:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790557696/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828772.034495:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790557056/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828772.034552:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828772.034553:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828772.034572:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828772.034574:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828772.034580:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5408 00000100:00080000:3.0:1787828772.034584:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828772.034587:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828772.034591:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828772.034601:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790557056:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787828772.034603:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828772.034628:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790557824/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828772.034640:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828772.034643:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828772.034647:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828772.034654:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5408 00000100:00100000:0.0:1787828772.034658:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790557184:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828772.034667:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828772.034738:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828772.034758:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828772/real 1787828772] req@ffff9e680a2d5a40 x1874668790557184/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828772.034775:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790557184/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828772.034793:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828772.034798:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828772.034802:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790557184:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828772.034814:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790557312:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787828772.034834:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828772.034843:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828772.034845:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828772.034869:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790557952/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828772.034876:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828772/real 1787828772] req@ffff9e680a2d43c0 x1874668790557312/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828772.034956:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790557312/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828772.034969:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828772.034974:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828772.034979:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790557312:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828772.034987:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790557440:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828772.035011:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828772.035023:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828772/real 1787828772] req@ffff9e680a2d6580 x1874668790557440/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828772.035031:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790557440/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828772.035038:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828772.035040:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828772.035043:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790557440:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828772.035048:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790557568:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828772.035061:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828772.035068:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828772/real 1787828772] req@ffff9e680a2d52c0 x1874668790557568/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828772.035075:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790557568/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828772.035083:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828772.035087:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828772.035090:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790557568:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828772.035096:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790557696:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828772.035117:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828772.035129:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828772/real 1787828772] req@ffff9e680a2d70c0 x1874668790557696/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828772.035141:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790557696/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828772.035149:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828772.035153:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828772.035156:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790557696:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828772.035163:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790557824:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828772.035183:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828772.035200:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828772/real 1787828772] req@ffff9e680a2d7c00 x1874668790557824/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828772.035213:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790557824/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828772.035221:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828772.035224:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828772.035227:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790557824:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828772.035270:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790557952:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828772.035294:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828772.035307:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828772/real 1787828772] req@ffff9e680a2d4f00 x1874668790557952/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828772.035317:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790557952/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828772.035325:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828772.035329:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828772.035332:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790557952:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828772.545783:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828772.737873:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828773.569802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828773.761871:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828774.593787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828774.785844:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828775.617788:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828775.809936:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828776.641785:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828776.833821:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828777.153784:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828777.153792:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828777.153797:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828777.153805:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5413 00000100:00080000:2.0:1787828777.153814:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828777.153820:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828777.153852:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790558080/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828777.153873:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828777.153875:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828777.153876:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828777.153879:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5413 00000100:00080000:2.0:1787828777.153882:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828777.153885:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828777.153893:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828777.153902:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790558208/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828777.153907:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790558080:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828777.153913:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828777.153916:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828777.153917:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828777.153920:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5413 00000100:00080000:2.0:1787828777.153922:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828777.153924:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828777.153928:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828777.153935:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790558336/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828777.153946:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828777.153949:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828777.153950:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828777.153952:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5413 00000100:00080000:2.0:1787828777.153955:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828777.153958:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828777.153960:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828777.153968:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828777.153969:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790558464/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828777.153978:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828777.153981:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828777.153982:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828777.153985:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5413 00000100:00080000:2.0:1787828777.153988:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828777.153990:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828777.153993:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828777.154002:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790558592/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828777.154007:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828777.154009:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828777.154010:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828777.154013:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5413 00000100:00080000:2.0:1787828777.154016:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828777.154022:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828777/real 1787828777] req@ffff9e6804422d00 x1874668790558080/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828777.154022:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828777.154027:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828777.154038:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790558720/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828777.154047:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828777.154049:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828777.154051:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828777.154054:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790558080/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828777.154056:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5413 00000100:00080000:2.0:1787828777.154060:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828777.154065:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828777.154071:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787828777.154073:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787828777.154079:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790558848/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828777.154080:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828777.154088:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790558080:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828777.154095:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828777.154098:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828777.154100:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828777.154103:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5413 00000100:00080000:2.0:1787828777.154106:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828777.154109:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828777.154111:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828777.154117:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790558208:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828777.154120:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790558976/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828777.154132:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828777.154135:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828777.154137:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828777.154140:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5413 00000800:00000100:0.0:1787828777.154145:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828777.154145:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828777.154152:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828777.154156:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828777.154163:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828777/real 1787828777] req@ffff9e6804422940 x1874668790558208/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828777.154166:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790559104/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828777.154193:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790558208/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828777.154201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828777.154204:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828777.154222:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790558208:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828777.154227:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790558336:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828777.154240:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828777.154247:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828777/real 1787828777] req@ffff9e6804421e00 x1874668790558336/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828777.154255:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790558336/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828777.154260:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828777.154262:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828777.154264:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790558336:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828777.154268:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790558464:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828777.154279:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828777.154284:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828777/real 1787828777] req@ffff9e6804420f00 x1874668790558464/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828777.154291:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790558464/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828777.154297:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828777.154298:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828777.154300:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790558464:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828777.154304:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790558592:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828777.154313:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828777.154318:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828777/real 1787828777] req@ffff9e6804421680 x1874668790558592/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828777.154325:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790558592/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828777.154330:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828777.154332:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828777.154334:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790558592:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828777.154337:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790558720:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828777.154346:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828777.154352:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828777/real 1787828777] req@ffff9e6804423480 x1874668790558720/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828777.154359:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790558720/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828777.154365:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828777.154367:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828777.154369:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790558720:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828777.154372:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790558848:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828777.154382:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828777.154387:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828777/real 1787828777] req@ffff9e68044221c0 x1874668790558848/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828777.154395:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790558848/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828777.154400:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828777.154401:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828777.154403:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790558848:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828777.154415:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790558976:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828777.154425:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828777.154430:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828777/real 1787828777] req@ffff9e6804423840 x1874668790558976/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828777.154436:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790558976/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828777.154446:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828777.154447:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828777.154449:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790558976:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828777.154452:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790559104:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828777.154467:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828777.154473:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828777/real 1787828777] req@ffff9e6804422580 x1874668790559104/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828777.154479:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790559104/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828777.154484:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828777.154485:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828777.154487:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790559104:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828777.665816:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828777.857904:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828778.689756:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828778.881867:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828779.713824:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828779.905877:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828780.737756:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828780.929867:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828781.761843:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828781.953833:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828782.273835:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828782.273848:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828782.273853:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828782.273864:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5418 00000100:00080000:1.0:1787828782.273876:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828782.273884:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828782.273930:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790559232/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828782.273961:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828782.273965:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828782.273967:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828782.273970:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5418 00000100:00080000:1.0:1787828782.273973:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828782.273976:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828782.273987:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828782.273997:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790559360/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828782.274009:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790559232:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828782.274012:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828782.274027:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828782.274030:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828782.274034:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5418 00000100:00080000:1.0:1787828782.274036:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828782.274039:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828782.274042:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828782.274054:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790559488/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828782.274061:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828782.274064:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828782.274065:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828782.274068:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5418 00000100:00080000:1.0:1787828782.274083:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828782.274087:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787828782.274096:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1787828782.274100:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828782.274114:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790559616/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828782.274122:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828782.274125:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828782.274127:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828782.274131:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5418 00000100:00080000:1.0:1787828782.274136:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828782.274139:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828782.274144:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828782/real 1787828782] req@ffff9e6831aa8f00 x1874668790559232/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828782.274144:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828782.274159:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790559744/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828782.274167:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790559232/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828782.274191:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828782.274198:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828782.274200:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828782.274201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828782.274207:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5418 00000100:00080000:0.0:1787828782.274208:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828782.274214:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828782.274218:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790559232:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828782.274224:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828782.274234:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828782.274245:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790559872/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828782.274253:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790559360:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828782.274262:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828782.274265:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828782.274268:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828782.274272:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5418 00000100:00080000:1.0:1787828782.274279:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828782.274286:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787828782.274291:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1787828782.274299:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828782.274325:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828782/real 1787828782] req@ffff9e6831aaa580 x1874668790559360/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828782.274326:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790560000/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828782.274392:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828782.274393:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790559360/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828782.274407:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828782.274414:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828782.274429:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828782.274429:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5418 00000100:00080000:1.0:1787828782.274440:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828782.274442:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828782.274454:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790559360:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828782.274457:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828782.274471:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828782.274472:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790559488:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828782.274497:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790560128/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828782.274509:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828782.274522:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828782.274528:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828782/real 1787828782] req@ffff9e6831aa9680 x1874668790559488/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828782.274529:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828782.274536:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828782.274551:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790559488/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828782.274553:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5418 00000100:00080000:0.0:1787828782.274576:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828782.274577:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828782.274587:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828782.274588:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828782.274596:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790559488:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787828782.274597:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828782.274603:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790559616:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828782.274613:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790560256/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828782.274631:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828782.274652:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828782/real 1787828782] req@ffff9e6831aa9e00 x1874668790559616/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828782.274666:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790559616/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828782.274672:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828782.274674:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828782.274676:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790559616:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828782.274680:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790559744:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828782.274715:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828782.274724:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828782/real 1787828782] req@ffff9e6831aaa940 x1874668790559744/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828782.274731:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790559744/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828782.274737:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828782.274739:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828782.274741:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790559744:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828782.274755:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790559872:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828782.274767:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828782.274774:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828782/real 1787828782] req@ffff9e6831aa9a40 x1874668790559872/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828782.274781:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790559872/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828782.274792:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828782.274794:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828782.274797:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790559872:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828782.274801:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790560000:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828782.274811:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828782.274816:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828782/real 1787828782] req@ffff9e6831aabc00 x1874668790560000/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828782.274823:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790560000/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828782.274828:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828782.274830:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828782.274832:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790560000:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828782.274836:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790560128:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828782.274845:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828782.274850:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828782/real 1787828782] req@ffff9e6831aab480 x1874668790560128/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828782.274857:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790560128/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828782.274862:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828782.274864:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828782.274867:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790560128:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828782.274870:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790560256:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828782.274880:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828782.274885:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828782/real 1787828782] req@ffff9e6831aaad00 x1874668790560256/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828782.274891:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790560256/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828782.274896:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828782.274898:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828782.274900:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790560256:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828782.785751:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828782.977946:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828783.809785:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828784.001849:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828784.833808:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828785.025823:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828785.857771:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828786.049854:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828786.881814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828787.073907:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828787.393838:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828787.393851:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828787.393856:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828787.393868:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5424 00000100:00080000:2.0:1787828787.393891:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828787.393899:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828787.393936:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790560384/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828787.393964:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828787.393969:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828787.393971:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828787.393993:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5424 00000100:00080000:2.0:1787828787.393997:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828787.394002:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828787.394013:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828787.394015:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790560384:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787828787.394031:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790560512/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828787.394042:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828787.394045:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828787.394046:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828787.394049:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5424 00000100:00080000:2.0:1787828787.394070:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828787.394073:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828787.394078:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828787.394091:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790560640/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828787.394096:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828787.394108:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828787.394112:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828787.394114:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828787.394119:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5424 00000100:00080000:2.0:1787828787.394122:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828787.394126:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828787.394130:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828787.394162:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828787/real 1787828787] req@ffff9e6804420b40 x1874668790560384/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828787.394165:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790560768/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828787.394221:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828787.394226:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787828787.394231:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790560384/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828787.394233:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828787.394244:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5424 00000100:00080000:2.0:1787828787.394256:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828787.394263:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828787.394268:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787828787.394273:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828787.394282:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790560384:192.168.203.109@tcp:250:kworker.0 00000008:00100000:2.0:1787828787.394284:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828787.394315:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790560512:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828787.394339:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790560896/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828787.394356:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828787.394360:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:0.0:1787828787.394366:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828787.394366:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828787.394374:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5424 00000100:00080000:2.0:1787828787.394380:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828787.394402:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828787/real 1787828787] req@ffff9e68044230c0 x1874668790560512/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828787.394403:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828787.394414:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828787.394419:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790560512/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828787.394429:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828787.394433:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828787.394434:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790561024/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828787.394437:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790560512:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828787.394464:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790560640:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828787.394479:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828787.394484:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828787.394487:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828787.394494:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5424 00000800:00000100:0.0:1787828787.394500:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828787.394507:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828787.394533:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828787/real 1787828787] req@ffff9e68044212c0 x1874668790560640/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828787.394535:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828787.394549:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828787.394554:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790560640/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828787.394564:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828787.394567:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828787.394574:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790560640:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828787.394576:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790561152/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828787.394599:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790560768:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787828787.394612:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828787.394615:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828787.394617:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787828787.394624:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828787.394624:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5424 00000100:00080000:2.0:1787828787.394631:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828787.394635:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828787.394639:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828787/real 1787828787] req@ffff9e6804420780 x1874668790560768/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828787.394642:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828787.394671:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790560768/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828787.394682:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828787.394686:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828787.394689:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790560768:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828787.394693:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790561280/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828787.394705:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790560896:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828787.394724:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828787.394734:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828787/real 1787828787] req@ffff9e6804422580 x1874668790560896/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828787.394767:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828787.394770:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790560896/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828787.394773:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828787.394778:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828787.394788:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5424 00000100:00080000:2.0:1787828787.394793:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828787.394814:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828787.394820:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787828787.394829:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828787.394835:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828787.394836:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790561408/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828787.394839:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790560896:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828787.394851:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790561024:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828787.394873:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828787.394886:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828787/real 1787828787] req@ffff9e6804423840 x1874668790561024/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828787.394898:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790561024/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828787.394907:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828787.394911:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828787.394915:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790561024:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828787.394921:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790561152:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828787.394939:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828787.394947:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828787/real 1787828787] req@ffff9e68044221c0 x1874668790561152/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828787.394955:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790561152/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828787.394960:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828787.394962:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828787.394964:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790561152:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828787.394974:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790561280:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828787.394984:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828787.394990:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828787/real 1787828787] req@ffff9e6804423480 x1874668790561280/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828787.394997:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790561280/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828787.395002:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828787.395003:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828787.395005:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790561280:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828787.395008:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790561408:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828787.395018:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828787.395042:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828787/real 1787828787] req@ffff9e6804421680 x1874668790561408/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828787.395051:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790561408/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828787.395058:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828787.395060:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828787.395062:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790561408:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828787.905784:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828788.097927:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828788.929834:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828789.122022:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828789.953772:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828790.146527:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828790.977805:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828791.169877:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828792.001786:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828792.193879:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828792.513834:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828792.513843:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828792.513848:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828792.513858:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5429 00000100:00080000:0.0:1787828792.513868:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828792.513887:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828792.513924:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790561536/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828792.513954:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828792.513958:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828792.513961:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828792.513966:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5429 00000100:00080000:0.0:1787828792.513971:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828792.513976:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828792.513983:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790561536:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1787828792.513986:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828792.514004:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790561664/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828792.514019:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828792.514033:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828792.514035:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828792.514040:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5429 00000100:00080000:0.0:1787828792.514045:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828792.514050:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787828792.514051:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787828792.514058:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828792.514070:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790561792/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828792.514078:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828792.514080:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828792.514082:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828792.514084:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5429 00000100:00080000:0.0:1787828792.514087:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828792.514093:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828792.514094:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828792/real 1787828792] req@ffff9e6803e98f00 x1874668790561536/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828792.514097:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828792.514128:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790561920/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828792.514131:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790561536/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828792.514155:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828792.514156:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828792.514162:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828792.514164:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787828792.514165:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828792.514170:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790561536:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828792.514172:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5429 00000100:00080000:0.0:1787828792.514178:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828792.514182:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828792.514185:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828792.514195:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790561664:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828792.514196:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790562048/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828792.514209:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828792.514212:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828792.514213:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828792.514216:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5429 00000100:00080000:0.0:1787828792.514219:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828792.514223:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828792.514226:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828792.514231:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828792.514238:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828792/real 1787828792] req@ffff9e6803e99e00 x1874668790561664/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828792.514242:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790562176/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828792.514263:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828792.514266:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790561664/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828792.514267:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828792.514270:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828792.514278:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5429 00000100:00080000:0.0:1787828792.514288:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828792.514291:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828792.514292:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787828792.514298:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:0.0:1787828792.514299:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828792.514304:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790561664:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828792.514311:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790562304/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828792.514312:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790561792:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828792.514331:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828792.514335:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828792.514337:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828792.514340:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828792.514344:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5429 00000100:00100000:2.0:1787828792.514347:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828792/real 1787828792] req@ffff9e6803e9a940 x1874668790561792/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828792.514349:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828792.514357:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828792.514365:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828792.514369:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790561792/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828792.514376:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828792.514378:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828792.514380:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790561792:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828792.514385:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790562432/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828792.514387:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790561920:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787828792.514400:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828792.514403:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828792.514404:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828792.514407:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5429 00000100:00080000:0.0:1787828792.514411:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828792.514416:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787828792.514418:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787828792.514420:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828792.514430:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790562560/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828792.514437:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828792/real 1787828792] req@ffff9e6803e9ad00 x1874668790561920/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828792.514453:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790561920/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828792.514459:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828792.514462:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828792.514464:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790561920:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828792.514475:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790562048:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828792.514490:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828792.514497:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828792/real 1787828792] req@ffff9e6803e98f00 x1874668790562048/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828792.514504:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790562048/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828792.514514:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828792.514516:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828792.514518:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790562048:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828792.514522:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790562176:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828792.514536:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828792.514546:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828792/real 1787828792] req@ffff9e6803e992c0 x1874668790562176/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828792.514570:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790562176/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828792.514580:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828792.514584:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828792.514587:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790562176:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828792.514592:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790562304:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828792.514607:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828792.514627:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828792/real 1787828792] req@ffff9e6803e99680 x1874668790562304/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828792.514635:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790562304/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828792.514640:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828792.514642:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828792.514644:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790562304:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828792.514648:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790562432:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828792.514659:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828792.514665:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828792/real 1787828792] req@ffff9e6803e9a580 x1874668790562432/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828792.514672:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790562432/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828792.514677:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828792.514679:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828792.514680:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790562432:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828792.514684:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790562560:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828792.514709:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828792.514726:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828792/real 1787828792] req@ffff9e6803e9b480 x1874668790562560/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828792.514734:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790562560/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828792.514739:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828792.514741:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828792.514743:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790562560:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828793.025814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828793.217891:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828794.049835:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828794.242804:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828795.073782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828795.265849:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828796.097810:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828796.289895:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828797.121800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828797.313837:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828797.633798:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828797.633808:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828797.633814:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828797.633824:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5434 00000100:00080000:1.0:1787828797.633836:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828797.633844:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828797.633883:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790562688/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828797.633912:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828797.633915:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828797.633917:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828797.633920:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5434 00000100:00080000:1.0:1787828797.633923:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828797.633926:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828797.633933:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828797.633943:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790562816/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828797.633951:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828797.633953:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828797.633955:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828797.633957:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5434 00000100:00080000:1.0:1787828797.633959:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828797.633964:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828797.633966:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790562688:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787828797.633967:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828797.633977:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790562944/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828797.633984:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828797.633986:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828797.633987:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828797.633989:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5434 00000100:00080000:1.0:1787828797.633991:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828797.633993:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828797.633996:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828797.634004:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790563072/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828797.634010:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828797.634012:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828797.634013:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828797.634016:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5434 00000100:00080000:1.0:1787828797.634019:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828797.634022:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828797.634025:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828797.634032:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790563200/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828797.634040:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828797.634043:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828797.634050:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828797.634052:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828797.634057:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5434 00000100:00080000:1.0:1787828797.634079:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828797.634083:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828797.634086:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828797.634096:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790563328/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828797.634103:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828797/real 1787828797] req@ffff9e6831aa8000 x1874668790562688/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828797.634105:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828797.634112:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828797.634114:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828797.634124:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5434 00000100:00080000:1.0:1787828797.634131:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828797.634135:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828797.634140:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790562688/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828797.634143:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828797.634158:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828797.634160:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790563456/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828797.634166:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828797.634175:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790562688:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828797.634191:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828797.634191:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790562816:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828797.634198:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828797.634200:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828797.634204:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5434 00000100:00080000:1.0:1787828797.634207:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828797.634210:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828797.634214:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828797.634226:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790563584/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828797.634227:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828797.634238:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828797.634241:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828797.634242:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828797.634245:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5434 00000100:00080000:1.0:1787828797.634249:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828797.634249:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828797/real 1787828797] req@ffff9e6831aaa1c0 x1874668790562816/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828797.634253:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828797.634259:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828797.634269:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790563712/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828797.634276:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790562816/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828797.634289:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828797.634294:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828797.634298:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790562816:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828797.634323:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790562944:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828797.634350:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828797.634364:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828797/real 1787828797] req@ffff9e6831aa8780 x1874668790562944/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828797.634376:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790562944/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828797.634385:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828797.634388:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828797.634391:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790562944:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828797.634398:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790563072:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828797.634420:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828797.634433:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828797/real 1787828797] req@ffff9e6831aa8b40 x1874668790563072/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828797.634446:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790563072/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828797.634455:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828797.634458:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828797.634462:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790563072:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828797.634468:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790563200:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828797.634488:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828797.634499:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828797/real 1787828797] req@ffff9e6831aaad00 x1874668790563200/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828797.634518:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790563200/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828797.634526:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828797.634529:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828797.634533:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790563200:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828797.634539:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790563328:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828797.634557:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828797.634568:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828797/real 1787828797] req@ffff9e6831aab480 x1874668790563328/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828797.634580:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790563328/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828797.634589:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828797.634593:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828797.634596:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790563328:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828797.634602:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790563456:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828797.634620:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828797.634632:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828797/real 1787828797] req@ffff9e6831aabc00 x1874668790563456/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828797.634645:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790563456/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828797.634659:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828797.634663:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828797.634667:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790563456:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828797.634673:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790563584:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828797.634718:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828797.634732:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828797/real 1787828797] req@ffff9e6831aa9a40 x1874668790563584/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828797.634745:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790563584/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828797.634754:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828797.634758:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828797.634761:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790563584:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828797.634768:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790563712:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828797.634800:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828797.634813:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828797/real 1787828797] req@ffff9e6831aaa940 x1874668790563712/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828797.634826:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790563712/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828797.634834:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828797.634837:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828797.634842:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790563712:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828798.145799:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828798.337981:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828799.169818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828799.361875:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828800.193781:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828800.385891:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828801.217813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828801.409876:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828802.241767:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828802.433887:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828802.753832:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828802.753840:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828802.753844:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828802.753853:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5439 00000100:00080000:3.0:1787828802.753863:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828802.753869:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828802.753898:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790563840/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828802.753918:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828802.753920:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828802.753922:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828802.753925:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5439 00000100:00080000:3.0:1787828802.753927:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828802.753929:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828802.753937:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828802.753947:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790563968/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828802.753959:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790563840:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828802.753962:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828802.753978:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828802.753980:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828802.753985:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5439 00000100:00080000:3.0:1787828802.753987:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828802.753990:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828802.753993:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828802.754002:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790564096/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828802.754009:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828802.754011:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828802.754013:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828802.754016:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5439 00000100:00080000:3.0:1787828802.754019:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828802.754022:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828802.754025:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828802.754046:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790564224/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828802.754052:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828802.754057:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828802.754060:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828802.754062:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828802.754103:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828802/real 1787828802] req@ffff9e680a2d61c0 x1874668790563840/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828802.754124:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790563840/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828802.754132:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828802.754135:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828802.754140:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790563840:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828802.754142:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5439 00000100:00080000:3.0:1787828802.754148:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828802.754152:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828802.754156:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828802.754166:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790563968:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828802.754169:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790564352/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828802.754179:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828802.754182:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828802.754184:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787828802.754191:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828802.754201:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828802/real 1787828802] req@ffff9e680a2d4b40 x1874668790563968/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828802.754210:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790563968/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828802.754216:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828802.754218:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828802.754221:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790563968:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828802.754233:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790564096:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828802.754249:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828802.754257:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828802/real 1787828802] req@ffff9e680a2d7480 x1874668790564096/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828802.754266:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790564096/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828802.754276:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828802.754279:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5439 00000100:00080000:0.0:1787828802.754282:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828802.754286:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828802.754287:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790564096:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828802.754294:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828802.754301:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828802.754302:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790564224:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828802.754321:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790564480/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828802.754327:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828802.754343:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828802/real 1787828802] req@ffff9e680a2d5e00 x1874668790564224/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828802.754359:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790564224/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828802.754368:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828802.754372:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828802.754376:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790564224:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828802.754396:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790564352:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828802.754415:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828802.754424:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828802/real 1787828802] req@ffff9e680a2d6d00 x1874668790564352/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828802.754440:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828802.754444:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790564352/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828802.754445:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828802.754448:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828802.754456:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5439 00000100:00080000:0.0:1787828802.754460:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828802.754461:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828802.754467:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828802.754469:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828802.754474:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790564352:192.168.203.109@tcp:8:kworker.0 00000008:00100000:3.0:1787828802.754477:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828802.754487:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790564480:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828802.754492:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790564608/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828802.754506:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828802.754506:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828802.754512:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828802.754514:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828802.754519:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5439 00000100:00100000:0.0:1787828802.754520:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828802/real 1787828802] req@ffff9e680a2d61c0 x1874668790564480/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828802.754522:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828802.754530:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828802.754535:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828802.754536:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790564480/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828802.754545:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828802.754545:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790564736/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828802.754553:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828802.754560:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790564480:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828802.754565:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828802.754568:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828802.754570:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828802.754573:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790564608:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828802.754574:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5439 00000100:00080000:3.0:1787828802.754581:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828802.754584:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828802.754588:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828802.754595:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828802.754599:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790564864/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828802.754618:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828802/real 1787828802] req@ffff9e680a2d4f00 x1874668790564608/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828802.754626:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790564608/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828802.754632:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828802.754634:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828802.754636:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790564608:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828802.754640:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790564736:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828802.754651:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828802.754658:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828802/real 1787828802] req@ffff9e680a2d7c00 x1874668790564736/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828802.754665:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790564736/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828802.754670:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828802.754672:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828802.754674:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790564736:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828802.754681:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790564864:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828802.754691:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828802.754722:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828802/real 1787828802] req@ffff9e680a2d70c0 x1874668790564864/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828802.754731:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790564864/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828802.754737:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828802.754739:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828802.754741:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790564864:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828803.265768:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828803.457828:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828804.289776:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828804.481894:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828805.313834:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828805.505902:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828806.337799:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828806.529874:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828807.361772:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828807.553928:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828807.873807:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828807.873817:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828807.873822:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828807.873831:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5444 00000100:00080000:0.0:1787828807.873841:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828807.873847:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828807.873881:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790564992/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828807.873907:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828807.873910:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828807.873912:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828807.873915:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5444 00000100:00080000:0.0:1787828807.873918:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828807.873922:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828807.873931:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828807.873943:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790565120/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828807.873950:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790564992:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828807.873960:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828807.873964:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828807.873966:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828807.873970:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5444 00000100:00080000:0.0:1787828807.873973:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828807.873976:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828807.873980:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828807.873991:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790565248/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828807.874015:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828807.874017:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828807.874019:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828807.874024:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5444 00000100:00080000:0.0:1787828807.874028:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828807.874035:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787828807.874035:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787828807.874045:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828807.874058:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790565376/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828807.874066:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828807.874069:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828807.874071:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828807.874074:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5444 00000100:00080000:0.0:1787828807.874077:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828807.874079:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828807/real 1787828807] req@ffff9e6803e9b840 x1874668790564992/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828807.874084:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828807.874092:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828807.874108:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790565504/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828807.874111:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790564992/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828807.874135:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828807.874140:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828807.874142:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828807.874147:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787828807.874149:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828807.874152:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790564992:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828807.874162:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5444 00000100:00080000:0.0:1787828807.874170:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828807.874175:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828807.874181:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790565120:192.168.203.109@tcp:38:kworker.0 00000008:00100000:0.0:1787828807.874184:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828807.874205:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790565632/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828807.874210:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828807.874223:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828807/real 1787828807] req@ffff9e6803e9a1c0 x1874668790565120/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828807.874248:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828807.874252:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828807.874255:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790565120/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828807.874256:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828807.874267:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5444 00000100:00080000:0.0:1787828807.874276:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828807.874277:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828807.874284:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787828807.874287:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828807.874309:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790565120:192.168.203.109@tcp:38:kworker.0 00000008:00100000:0.0:1787828807.874313:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828807.874318:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790565248:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828807.874330:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790565760/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828807.874339:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828807.874342:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828807.874344:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828807.874347:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5444 00000100:00080000:0.0:1787828807.874351:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828807.874355:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787828807.874361:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787828807.874363:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828807.874375:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828807/real 1787828807] req@ffff9e6803e98b40 x1874668790565248/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828807.874377:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790565888/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828807.874409:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828807.874411:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790565248/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828807.874415:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828807.874419:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828807.874429:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5444 00000100:00080000:2.0:1787828807.874430:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828807.874436:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828807.874438:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787828807.874443:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828807.874446:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790565248:192.168.203.109@tcp:38:kworker.0 00000008:00100000:0.0:1787828807.874449:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828807.874455:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790565376:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828807.874462:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790566016/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828807.874479:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828807.874491:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828807/real 1787828807] req@ffff9e6803e98780 x1874668790565376/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828807.874501:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790565376/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828807.874507:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828807.874509:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828807.874513:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790565376:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828807.874517:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790565504:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828807.874529:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828807.874535:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828807/real 1787828807] req@ffff9e6803e9bc00 x1874668790565504/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828807.874544:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790565504/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828807.874550:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828807.874552:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828807.874554:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790565504:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828807.874567:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790565632:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828807.874579:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828807.874586:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828807/real 1787828807] req@ffff9e6803e9b840 x1874668790565632/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828807.874595:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790565632/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828807.874602:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828807.874604:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828807.874606:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790565632:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828807.874610:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790565760:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828807.874622:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828807.874629:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828807/real 1787828807] req@ffff9e6803e9b480 x1874668790565760/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828807.874637:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790565760/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828807.874643:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828807.874645:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828807.874647:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790565760:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828807.874667:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790565888:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828807.874685:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828807.874697:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828807/real 1787828807] req@ffff9e6803e9a580 x1874668790565888/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828807.874735:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790565888/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828807.874746:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828807.874750:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828807.874753:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790565888:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828807.874761:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790566016:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828807.874784:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828807.874807:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828807/real 1787828807] req@ffff9e6803e99680 x1874668790566016/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828807.874817:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790566016/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828832 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828807.874839:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828807.874841:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828807.874844:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790566016:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828808.385791:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828808.577938:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828809.409871:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828809.601868:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828810.433847:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828810.626006:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828811.457796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828811.650042:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828812.481879:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828812.673918:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828812.993876:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828812.993885:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828812.993889:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828812.993898:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5449 00000100:00080000:3.0:1787828812.993908:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828812.993916:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828812.993977:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790566144/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828812.994002:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828812.994005:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828812.994007:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828812.994010:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5449 00000100:00080000:3.0:1787828812.994014:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828812.994018:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828812.994027:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828812.994040:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790566272/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828812.994060:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828812.994065:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828812.994066:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790566144:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828812.994067:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828812.994073:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5449 00000100:00080000:3.0:1787828812.994080:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828812.994083:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828812.994087:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828812.994096:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790566400/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828812.994104:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828812.994108:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828812.994109:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828812.994112:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5449 00000100:00080000:3.0:1787828812.994114:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828812.994117:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828812.994120:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828812.994128:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790566528/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828812.994136:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828812.994138:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828812.994140:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828812.994143:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5449 00000100:00080000:3.0:1787828812.994146:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828812.994150:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828812.994153:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828812.994160:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828812.994174:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790566656/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828812.994186:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828812.994190:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828812.994193:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828812.994199:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5449 00000100:00100000:2.0:1787828812.994205:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828812/real 1787828812] req@ffff9e680a2d52c0 x1874668790566144/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828812.994207:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828812.994218:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828812.994230:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828812.994240:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790566144/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828812.994251:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828812.994257:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790566784/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828812.994258:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828812.994267:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790566144:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828812.994285:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828812.994289:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828812.994292:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828812.994296:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790566272:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828812.994308:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5449 00000100:00080000:3.0:1787828812.994314:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828812.994318:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828812.994323:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787828812.994327:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828812.994341:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790566912/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828812.994344:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828812/real 1787828812] req@ffff9e680a2d6580 x1874668790566272/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828812.994366:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828812.994370:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787828812.994372:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790566272/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828812.994373:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828812.994382:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5449 00000100:00080000:3.0:1787828812.994389:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828812.994395:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787828812.994396:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828812.994399:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828812.994403:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828812.994409:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790566272:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828812.994413:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790567040/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828812.994420:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790566400:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828812.994436:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828812.994439:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828812.994441:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828812.994445:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828812.994448:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5449 00000100:00080000:3.0:1787828812.994454:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828812.994458:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828812.994461:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828812.994471:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828812/real 1787828812] req@ffff9e680a2d43c0 x1874668790566400/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828812.994472:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790567168/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828812.994515:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790566400/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828812.994523:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828812.994526:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828812.994529:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790566400:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828812.994533:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790566528:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828812.994548:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828812.994558:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828812/real 1787828812] req@ffff9e680a2d5a40 x1874668790566528/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828812.994567:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790566528/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828812.994575:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828812.994579:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828812.994584:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790566528:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828812.994591:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790566656:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828812.994614:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828812.994628:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828812/real 1787828812] req@ffff9e680a2d4780 x1874668790566656/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828812.994643:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790566656/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828812.994653:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828812.994656:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828812.994660:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790566656:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828812.994668:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790566784:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828812.994691:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828812.994739:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828812/real 1787828812] req@ffff9e680a2d4000 x1874668790566784/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828812.994757:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790566784/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828812.994771:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828812.994776:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828812.994781:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790566784:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828812.994808:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790566912:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828812.994831:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828812.994842:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828812/real 1787828812] req@ffff9e680a2d52c0 x1874668790566912/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828812.994851:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790566912/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828812.994857:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828812.994860:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828812.994862:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790566912:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828812.994866:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790567040:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828812.994878:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828812.994884:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828812/real 1787828812] req@ffff9e680a2d70c0 x1874668790567040/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828812.994892:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790567040/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828812.994898:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828812.994900:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828812.994902:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790567040:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828812.994906:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790567168:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828812.994917:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828812.994923:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828812/real 1787828812] req@ffff9e680a2d7c00 x1874668790567168/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828812.994931:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790567168/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828837 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828812.994938:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828812.994940:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828812.994943:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790567168:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828813.505832:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828813.697838:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828814.529897:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828814.721922:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828815.553815:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828815.746011:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828816.577870:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828816.769929:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828817.601801:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828817.793876:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828818.113769:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828818.113777:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828818.113780:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828818.113788:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5454 00000100:00080000:3.0:1787828818.113797:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828818.113803:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828818.113835:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790567296/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828818.113857:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828818.113859:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828818.113861:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828818.113865:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5454 00000100:00080000:3.0:1787828818.113867:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828818.113870:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828818.113878:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828818.113887:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790567424/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828818.113894:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790567296:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828818.113898:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828818.113902:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828818.113903:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828818.113906:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5454 00000100:00080000:3.0:1787828818.113908:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828818.113911:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828818.113914:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828818.113934:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790567552/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828818.113942:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828818.113944:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828818.113945:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828818.113948:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5454 00000100:00080000:3.0:1787828818.113958:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828818.113961:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828818.113964:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828818.113972:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828818.113974:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790567680/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828818.113982:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828818.113984:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828818.113986:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828818.113988:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5454 00000100:00080000:3.0:1787828818.113991:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828818.113993:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828818.113996:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828818.114003:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790567808/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828818.114009:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828818.114012:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828818/real 1787828818] req@ffff9e680a2d4f00 x1874668790567296/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828818.114013:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828818.114016:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828818.114022:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5454 00000100:00080000:3.0:1787828818.114027:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828818.114031:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828818.114035:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828818.114039:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790567296/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828818.114046:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790567936/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828818.114050:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828818.114057:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828818.114058:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828818.114063:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787828818.114066:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790567296:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828818.114067:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828818.114076:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5454 00000100:00080000:3.0:1787828818.114079:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828818.114081:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828818.114084:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828818.114091:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790568064/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828818.114094:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790567424:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828818.114108:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828818.114111:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828818.114112:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828818.114116:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5454 00000100:00080000:3.0:1787828818.114119:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787828818.114122:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828818.114125:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828818.114130:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828818.114137:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828818/real 1787828818] req@ffff9e680a2d61c0 x1874668790567424/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828818.114141:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790568192/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828818.114156:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828818.114158:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790567424/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828818.114160:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828818.114162:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828818.114169:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5454 00000100:00080000:3.0:1787828818.114175:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828818.114176:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828818.114178:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828818.114184:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828818.114186:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828818.114192:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790568320/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828818.114195:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790567424:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828818.114211:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790567552:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828818.114236:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828818.114245:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828818/real 1787828818] req@ffff9e680a2d6d00 x1874668790567552/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828818.114253:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790567552/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828818.114259:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828818.114261:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828818.114263:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790567552:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828818.114267:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790567680:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828818.114283:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828818.114294:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828818/real 1787828818] req@ffff9e680a2d5e00 x1874668790567680/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828818.114307:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790567680/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828818.114317:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828818.114320:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828818.114324:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790567680:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828818.114328:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790567808:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828818.114343:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828818.114350:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828818/real 1787828818] req@ffff9e680a2d7480 x1874668790567808/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828818.114358:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790567808/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828818.114363:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828818.114365:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828818.114367:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790567808:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828818.114370:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790567936:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828818.114380:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828818.114385:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828818/real 1787828818] req@ffff9e680a2d4b40 x1874668790567936/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828818.114392:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790567936/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828818.114399:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828818.114400:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828818.114402:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790567936:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828818.114416:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790568064:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828818.114426:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828818.114433:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828818/real 1787828818] req@ffff9e680a2d4f00 x1874668790568064/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828818.114440:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790568064/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828818.114447:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828818.114450:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828818.114453:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790568064:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828818.114459:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790568192:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828818.114477:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828818.114488:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828818/real 1787828818] req@ffff9e680a2d7c00 x1874668790568192/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828818.114500:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790568192/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828818.114506:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828818.114508:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828818.114510:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790568192:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828818.114514:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790568320:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828818.114528:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828818.114534:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828818/real 1787828818] req@ffff9e680a2d70c0 x1874668790568320/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828818.114542:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790568320/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828818.114547:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828818.114549:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828818.114551:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790568320:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828818.625798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828818.817885:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828819.649814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828819.841869:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828820.673876:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828820.865981:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828821.697801:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828821.889885:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828822.721770:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828822.913885:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828823.233807:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828823.233817:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828823.233822:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828823.233832:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5459 00000100:00080000:3.0:1787828823.233844:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828823.233852:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828823.233902:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790568448/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828823.233946:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828823.233950:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828823.233952:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828823.233956:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5459 00000100:00080000:3.0:1787828823.233960:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828823.233965:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828823.233975:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828823.233991:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790568576/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828823.233998:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790568448:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828823.234009:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828823.234014:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828823.234016:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828823.234023:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5459 00000100:00080000:3.0:1787828823.234046:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828823.234052:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787828823.234057:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787828823.234059:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828823.234075:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790568704/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828823.234086:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828823.234090:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828823.234092:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828823.234094:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828823/real 1787828823] req@ffff9e680a2d52c0 x1874668790568448/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828823.234099:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5459 00000100:00080000:3.0:1787828823.234108:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828823.234114:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828823.234117:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790568448/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828823.234121:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828823.234144:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828823.234149:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828823.234153:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790568448:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828823.234157:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790568832/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828823.234169:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828823.234173:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828823.234175:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828823.234177:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790568576:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828823.234183:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5459 00000100:00080000:3.0:1787828823.234189:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828823.234194:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787828823.234202:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787828823.234212:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828823.234214:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828823/real 1787828823] req@ffff9e680a2d4000 x1874668790568576/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828823.234226:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790568576/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828823.234232:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790568960/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828823.234236:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828823.234244:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828823.234251:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790568576:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828823.234260:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790568704:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828823.234260:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828823.234269:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828823.234273:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828823.234279:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5459 00000800:00000100:2.0:1787828823.234283:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828823.234288:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828823.234294:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828823/real 1787828823] req@ffff9e680a2d4780 x1874668790568704/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828823.234298:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828823.234309:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828823.234313:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790568704/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828823.234321:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828823.234327:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828823.234329:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790569088/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828823.234330:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790568704:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828823.234342:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790568832:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787828823.234354:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828823.234359:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:2.0:1787828823.234362:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828823.234364:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828823.234373:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5459 00000100:00100000:2.0:1787828823.234374:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828823/real 1787828823] req@ffff9e680a2d5a40 x1874668790568832/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828823.234381:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828823.234387:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790568832/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828823.234391:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787828823.234401:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828823.234402:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787828823.234407:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828823.234411:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790568832:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828823.234424:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790569216/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828823.234427:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790568960:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787828823.234446:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828823.234447:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828823.234454:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828823.234457:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828823.234460:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828823/real 1787828823] req@ffff9e680a2d43c0 x1874668790568960/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828823.234465:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5459 00000100:00080000:3.0:1787828823.234475:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828823.234476:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790568960/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828823.234486:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787828823.234487:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828823.234495:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:3.0:1787828823.234497:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828823.234498:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790568960:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828823.234508:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790569088:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828823.234520:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790569344/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787828823.234525:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828823.234551:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828823.234553:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828823/real 1787828823] req@ffff9e680a2d6580 x1874668790569088/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828823.234559:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828823.234564:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828823.234572:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5459 00000100:00100000:2.0:1787828823.234578:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790569088/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828823.234579:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828823.234588:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828823.234597:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787828823.234600:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828823.234606:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828823.234613:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790569088:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828823.234614:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790569472/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828823.234627:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790569216:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828823.234647:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828823.234656:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828823/real 1787828823] req@ffff9e680a2d52c0 x1874668790569216/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828823.234663:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790569216/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828823.234669:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828823.234671:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828823.234673:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790569216:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828823.234676:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790569344:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828823.234704:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828823.234710:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828823/real 1787828823] req@ffff9e680a2d70c0 x1874668790569344/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828823.234717:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790569344/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828823.234722:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828823.234724:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828823.234726:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790569344:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828823.234734:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790569472:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828823.234768:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828823.234775:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828823/real 1787828823] req@ffff9e680a2d7c00 x1874668790569472/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828823.234782:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790569472/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828823.234788:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828823.234790:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828823.234792:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790569472:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828823.745828:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828823.937985:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828824.769792:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828824.961838:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828825.793801:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828825.986003:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828826.817805:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828827.010013:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828827.841792:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828828.033832:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828828.353824:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828828.353833:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828828.353837:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828828.353845:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5465 00000100:00080000:0.0:1787828828.353867:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828828.353890:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828828.353923:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790569600/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828828.353945:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828828.353948:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828828.353950:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828828.353952:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5465 00000100:00080000:0.0:1787828828.353955:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828828.353958:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828828.353965:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828828.353974:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790569728/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828828.353983:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828828.353988:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828828.353991:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828828.353991:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790569600:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828828.354000:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5465 00000100:00080000:0.0:1787828828.354003:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828828.354005:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828828.354008:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828828.354018:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790569856/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828828.354025:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828828.354027:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828828.354029:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828828.354032:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5465 00000100:00080000:0.0:1787828828.354035:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828828.354037:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828828.354041:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828828.354048:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790569984/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828828.354054:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828828.354056:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828828.354059:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828828.354062:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828828.354066:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5465 00000100:00080000:0.0:1787828828.354069:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828828.354072:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828828.354076:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828828.354084:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790570112/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828828.354090:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828828.354092:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828828.354093:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828828.354096:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5465 00000100:00080000:0.0:1787828828.354099:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828828.354104:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828828/real 1787828828] req@ffff9e6803e992c0 x1874668790569600/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828828.354105:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828828.354112:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828828.354125:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790570240/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828828.354131:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828828.354134:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790569600/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828828.354135:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828828.354139:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828828.354143:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5465 00000100:00080000:0.0:1787828828.354145:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828828.354147:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828828.354150:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828828.354161:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790570368/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828828.354162:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828828.354171:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828828.354180:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790569600:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828828.354181:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828828.354189:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828828.354190:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828828.354194:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5465 00000100:00080000:0.0:1787828828.354196:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828828.354199:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828828.354201:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828828.354208:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790569728:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828828.354212:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790570496/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828828.354221:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828828.354224:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828828.354225:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828828.354229:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5465 00000100:00080000:0.0:1787828828.354232:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828828.354235:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828828.354239:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828828.354240:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828828.354252:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790570624/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828828.354258:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828828/real 1787828828] req@ffff9e6803e98f00 x1874668790569728/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828828.354271:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790569728/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828828.354279:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828828.354282:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828828.354285:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790569728:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828828.354289:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790569856:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828828.354317:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828828.354325:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828828/real 1787828828] req@ffff9e6803e9ad00 x1874668790569856/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828828.354333:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790569856/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828828.354338:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828828.354340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828828.354342:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790569856:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828828.354346:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790569984:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828828.354359:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828828.354365:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828828/real 1787828828] req@ffff9e6803e9a940 x1874668790569984/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828828.354373:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790569984/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828828.354378:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828828.354380:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828828.354383:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790569984:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828828.354386:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790570112:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828828.354400:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828828.354422:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828828/real 1787828828] req@ffff9e6803e99e00 x1874668790570112/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828828.354437:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790570112/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828828.354453:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828828.354456:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828828.354459:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790570112:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828828.354464:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790570240:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828828.354483:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828828.354495:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828828/real 1787828828] req@ffff9e6803e99680 x1874668790570240/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828828.354510:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790570240/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828828.354521:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828828.354524:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828828.354528:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790570240:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828828.354535:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790570368:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828828.354554:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828828.354564:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828828/real 1787828828] req@ffff9e6803e9a580 x1874668790570368/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828828.354582:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790570368/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828828.354588:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828828.354590:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828828.354592:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790570368:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828828.354607:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790570496:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828828.354636:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828828.354647:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828828/real 1787828828] req@ffff9e6803e9b480 x1874668790570496/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828828.354660:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790570496/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828828.354668:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828828.354672:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828828.354676:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790570496:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828828.354682:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790570624:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828828.354754:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828828.354763:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828828/real 1787828828] req@ffff9e6803e9b840 x1874668790570624/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828828.354772:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790570624/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828828.354778:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828828.354780:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828828.354783:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790570624:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828828.865786:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828829.057955:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828829.889872:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828830.081997:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828830.913913:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828831.106001:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828831.937857:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828832.129991:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828832.961794:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828833.153918:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828833.473852:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828833.473867:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828833.473873:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828833.473886:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5470 00000100:00080000:0.0:1787828833.473901:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828833.473912:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828833.473964:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790570752/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828833.474006:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828833.474011:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828833.474013:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828833.474019:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5470 00000100:00080000:0.0:1787828833.474024:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828833.474030:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828833.474043:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828833.474061:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790570880/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828833.474061:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790570752:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828833.474090:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828833.474096:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828833.474098:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828833.474105:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5470 00000100:00080000:0.0:1787828833.474111:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828833.474116:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828833.474122:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828833.474139:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790571008/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828833.474151:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828833.474155:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828833.474158:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828833.474166:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5470 00000100:00080000:0.0:1787828833.474173:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828833.474179:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828833.474185:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790570880:192.168.203.109@tcp:38:kworker.0 00000008:00100000:0.0:1787828833.474188:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828833.474214:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790571136/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828833.474224:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790571008:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828833.474248:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828833.474253:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828833.474256:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828833.474265:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790571136:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787828833.474266:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5470 00000100:00080000:0.0:1787828833.474280:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828833.474303:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828833.474311:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828833.474329:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790571264/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828833.474352:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828833.474357:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828833.474375:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828833.474381:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5470 00000100:00080000:0.0:1787828833.474386:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828833.474391:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790571264:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787828833.474400:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828833.474408:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828833.474426:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790571392/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828833.474456:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828833.474460:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828833.474463:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828833.474468:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5470 00000100:00080000:0.0:1787828833.474473:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828833.474478:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828833.474484:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790571392:192.168.203.109@tcp:38:kworker.0 00000008:00100000:0.0:1787828833.474488:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828833.474510:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790571520/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828833.474533:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828833.474537:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828833.474540:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828833.474545:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5470 00000100:00080000:0.0:1787828833.474550:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828833.474556:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790571520:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828833.474559:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828833.474572:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828833.474590:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790571648/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828833.474604:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828833.474608:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828833.474610:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828833.474616:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790571648:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787828833.474618:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5470 00000100:00080000:0.0:1787828833.474626:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828833.474634:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828833.474641:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828833.474657:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790571776/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828833.474691:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790571776:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828833.985834:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828834.177929:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828835.009801:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828835.201856:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828836.033780:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828836.225881:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787828836.546075:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:0.0:1787828836.546119:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787828836.546152:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828836.546214:0:2387:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:0.0:1787828836.546229:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828836.546241:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828836.546253:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787828836.546256:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828833/real 1787828836] req@ffff9e6803e99e00 x1874668790571776/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787828836.546266:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828836.546281:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787828836.546298:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790571776/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787828836.546299:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787828836.546313:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00080000:2.0:1787828836.546316:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828836.546330:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828836.546335:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790571776:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828836.546342:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828833/real 1787828836] req@ffff9e6803e99680 x1874668790571648/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828836.546356:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790571648/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828836.546365:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828836.546368:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828836.546372:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790571648:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828836.546379:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828833/real 1787828836] req@ffff9e6803e9a580 x1874668790571520/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828836.546391:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790571520/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828836.546399:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828836.546402:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828836.546406:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790571520:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828836.546416:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828833/real 1787828836] req@ffff9e6803e9b480 x1874668790571392/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828836.546428:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790571392/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828836.546439:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828836.546443:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828836.546446:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790571392:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828836.546453:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828833/real 1787828836] req@ffff9e6803e9b840 x1874668790571264/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828836.546466:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790571264/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828836.546475:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828836.546479:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828836.546482:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790571264:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828836.546489:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828833/real 1787828836] req@ffff9e6803e9a1c0 x1874668790571136/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828836.546502:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790571136/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828836.546511:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828836.546515:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828836.546519:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790571136:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828836.546526:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828833/real 1787828836] req@ffff9e6803e98780 x1874668790570880/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828836.546539:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790570880/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828836.546547:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828836.546551:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828836.546554:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790570880:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828836.546561:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828833/real 1787828836] req@ffff9e6803e98b40 x1874668790571008/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828836.546572:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790571008/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828836.546580:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828836.546584:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828836.546586:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790571008:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828836.546593:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828833/real 1787828836] req@ffff9e6803e9bc00 x1874668790570752/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828836.546605:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790570752/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828836.546621:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828836.546624:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828836.546627:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790570752:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787828837.057786:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828837.249893:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828838.081800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828838.273838:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828838.593891:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828838.593899:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828838.593903:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828838.593912:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5475 00000100:00080000:3.0:1787828838.593923:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828838.593930:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828838.593965:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790571904/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828838.593998:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828838.594002:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828838.594004:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828838.594009:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5475 00000100:00080000:3.0:1787828838.594015:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828838.594020:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828838.594040:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828838.594067:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790572032/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828838.594080:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790571904:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828838.594084:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828838.594092:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828838.594094:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828838.594099:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5475 00000100:00080000:3.0:1787828838.594105:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828838.594111:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828838.594129:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828838.594144:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790572160/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828838.594155:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828838.594159:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828838.594161:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787828838.594167:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828838.594169:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5475 00000100:00080000:3.0:1787828838.594176:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828838.594192:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828838.594198:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828838.594211:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790572288/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828838.594216:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828838/real 1787828838] req@ffff9e680a2d4f00 x1874668790571904/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828838.594234:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828838.594240:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828838.594243:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828838.594250:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5475 00000100:00100000:2.0:1787828838.594251:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790571904/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828838.594255:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828838.594264:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787828838.594272:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828838.594274:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828838.594289:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790572416/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828838.594300:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828838.594305:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828838.594309:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790571904:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828838.594313:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828838.594318:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828838.594324:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5475 00000100:00080000:3.0:1787828838.594329:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828838.594334:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828838.594339:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828838.594345:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790572032:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828838.594355:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790572544/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828838.594370:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828838.594374:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828838.594377:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828838.594383:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5475 00000100:00080000:3.0:1787828838.594388:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828838.594396:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787828838.594397:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787828838.594408:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828838.594423:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828838/real 1787828838] req@ffff9e680a2d4b40 x1874668790572032/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828838.594425:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790572672/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828838.594461:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828838.594466:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787828838.594470:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790572032/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828838.594470:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828838.594481:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5475 00000100:00080000:3.0:1787828838.594489:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828838.594492:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828838.594497:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787828838.594503:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:3.0:1787828838.594508:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828838.594511:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790572032:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828838.594523:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790572160:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828838.594528:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790572800/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828838.594547:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828838.594551:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828838.594554:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828838.594559:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5475 00000100:00080000:3.0:1787828838.594564:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828838.594570:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787828838.594577:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787828838.594579:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828838.594596:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790572928/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828838.594601:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828838/real 1787828838] req@ffff9e680a2d7480 x1874668790572160/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828838.594621:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790572160/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828838.594630:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828838.594634:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828838.594637:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790572160:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828838.594645:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790572288:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828838.594672:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828838.594690:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828838/real 1787828838] req@ffff9e680a2d5e00 x1874668790572288/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828838.594733:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790572288/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828838.594742:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828838.594746:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828838.594749:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790572288:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828838.594755:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790572416:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828838.594793:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828838.594807:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828838/real 1787828838] req@ffff9e680a2d6d00 x1874668790572416/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828838.594825:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790572416/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828838.594833:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828838.594836:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828838.594839:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790572416:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828838.594859:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790572544:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828838.594879:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828838.594891:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828838/real 1787828838] req@ffff9e680a2d61c0 x1874668790572544/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828838.594904:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790572544/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828838.594914:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828838.594918:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828838.594922:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790572544:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828838.594928:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790572672:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828838.594950:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828838.594963:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828838/real 1787828838] req@ffff9e680a2d4f00 x1874668790572672/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828838.594977:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790572672/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828838.594991:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828838.594994:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828838.594997:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790572672:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828838.595003:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790572800:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828838.595025:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828838.595038:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828838/real 1787828838] req@ffff9e680a2d7c00 x1874668790572800/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828838.595052:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790572800/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828838.595061:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828838.595064:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828838.595067:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790572800:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828838.595424:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790572928:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828838.595459:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828838.595474:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828838/real 1787828838] req@ffff9e680a2d70c0 x1874668790572928/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828838.595487:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790572928/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828838.595497:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828838.595501:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828838.595505:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790572928:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828839.105799:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828839.297870:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828840.129817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828840.321847:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828841.153780:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828841.345877:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828842.177799:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828842.369844:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828843.201819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828843.394022:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828843.713804:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828843.713812:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828843.713815:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828843.713823:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5480 00000100:00080000:0.0:1787828843.713832:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828843.713838:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828843.713870:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790573056/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828843.713893:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828843.713896:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828843.713897:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828843.713900:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5480 00000100:00080000:0.0:1787828843.713902:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828843.713905:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828843.713914:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828843.713925:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790573184/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828843.713927:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790573056:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828843.713941:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828843.713944:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828843.713945:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828843.713947:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5480 00000100:00080000:0.0:1787828843.713950:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828843.713952:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828843.713955:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828843.713963:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790573312/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828843.713970:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828843.713972:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828843.713973:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828843.713976:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5480 00000100:00080000:0.0:1787828843.713989:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828843.713992:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828843.713996:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828843.714002:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828843.714006:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790573440/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828843.714014:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828843.714016:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828843.714018:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828843.714020:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5480 00000100:00080000:0.0:1787828843.714023:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828843.714025:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828843.714028:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828843.714035:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790573568/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828843.714041:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828843.714043:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828843/real 1787828843] req@ffff9e6803e9a940 x1874668790573056/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828843.714044:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828843.714046:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828843.714057:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5480 00000100:00080000:0.0:1787828843.714062:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828843.714068:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828843.714072:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828843.714076:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790573056/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828843.714081:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790573696/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828843.714092:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828843.714098:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787828843.714099:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828843.714104:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790573056:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828843.714106:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828843.714111:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828843.714116:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5480 00000100:00080000:0.0:1787828843.714121:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828843.714126:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790573184:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828843.714128:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828843.714136:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828843.714149:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790573824/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828843.714158:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787828843.714161:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828843.714162:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828843.714165:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828843.714170:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5480 00000100:00080000:0.0:1787828843.714174:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787828843.714177:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828843/real 1787828843] req@ffff9e6803e9ad00 x1874668790573184/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828843.714181:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828843.714189:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828843.714194:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790573184/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828843.714200:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790573952/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828843.714205:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828843.714212:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828843.714213:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787828843.714220:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787828843.714221:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790573184:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828843.714223:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828843.714228:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5480 00000100:00100000:2.0:1787828843.714230:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790573312:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828843.714232:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828843.714238:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828843.714243:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787828843.714253:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828843.714254:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790574080/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828843.714264:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828843/real 1787828843] req@ffff9e6803e98f00 x1874668790573312/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828843.714273:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790573312/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828843.714279:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828843.714282:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828843.714284:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790573312:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828843.714288:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790573440:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828843.714299:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828843.714305:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828843/real 1787828843] req@ffff9e6803e992c0 x1874668790573440/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828843.714312:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790573440/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828843.714317:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828843.714319:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828843.714320:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790573440:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828843.714324:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790573568:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828843.714333:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828843.714339:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828843/real 1787828843] req@ffff9e6803e9bc00 x1874668790573568/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828843.714348:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790573568/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828843.714357:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828843.714360:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828843.714375:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790573568:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828843.714380:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790573696:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828843.714401:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828843.714413:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828843/real 1787828843] req@ffff9e6803e98b40 x1874668790573696/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828843.714421:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790573696/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828843.714428:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828843.714430:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828843.714432:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790573696:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828843.714445:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790573824:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787828843.714460:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828843.714471:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828843/real 1787828843] req@ffff9e6803e98780 x1874668790573824/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828843.714483:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790573824/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828843.714492:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828843.714495:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828843.714499:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790573824:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828843.714515:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790573952:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828843.714532:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828843.714540:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828843/real 1787828843] req@ffff9e6803e9a1c0 x1874668790573952/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828843.714547:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790573952/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828843.714553:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828843.714555:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828843.714557:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790573952:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828843.714560:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790574080:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828843.714570:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828843.714575:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828843/real 1787828843] req@ffff9e6803e9b840 x1874668790574080/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828843.714582:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790574080/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828843.714587:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828843.714588:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828843.714591:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790574080:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828844.225783:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828844.417859:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828845.249864:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828845.441897:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828846.273888:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828846.465902:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828847.297834:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828847.489967:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828848.321821:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828848.513877:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828848.833812:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828848.833824:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828848.833828:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828848.833840:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5485 00000100:00080000:2.0:1787828848.833852:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828848.833858:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828848.833891:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790574208/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828848.833916:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828848.833919:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828848.833920:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828848.833923:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5485 00000100:00080000:2.0:1787828848.833926:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828848.833929:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828848.833938:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828848.833948:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790574336/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828848.833952:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790574208:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828848.833970:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828848.833974:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828848.833977:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828848.833983:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5485 00000100:00080000:2.0:1787828848.833987:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828848.833992:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828848.833998:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828848.834012:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790574464/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828848.834015:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828848.834026:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828848.834029:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828848.834031:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828848.834036:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5485 00000100:00080000:2.0:1787828848.834039:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828848.834042:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828848.834047:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828848.834055:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828848/real 1787828848] req@ffff9e6804420f00 x1874668790574208/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828848.834058:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790574592/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828848.834081:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828848.834084:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787828848.834088:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790574208/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828848.834089:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828848.834096:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5485 00000100:00080000:0.0:1787828848.834106:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828848.834112:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787828848.834113:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828848.834116:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790574208:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828848.834120:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828848.834126:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828848.834136:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790574720/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828848.834141:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790574336:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828848.834152:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828848.834155:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828848.834158:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828848.834162:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5485 00000100:00080000:2.0:1787828848.834165:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828848.834166:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828848.834171:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828848.834177:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828848.834181:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828848/real 1787828848] req@ffff9e6804421e00 x1874668790574336/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828848.834190:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790574848/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828848.834196:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790574336/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828848.834205:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828848.834211:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828848.834212:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828848.834214:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828848.834218:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787828848.834222:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5485 00000100:00100000:0.0:1787828848.834226:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790574336:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828848.834227:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828848.834234:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828848.834235:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790574464:192.168.203.109@tcp:38:kworker.0 00000008:00100000:2.0:1787828848.834239:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828848.834259:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790574976/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828848.834260:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828848.834271:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828848.834272:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828848/real 1787828848] req@ffff9e6804422940 x1874668790574464/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828848.834281:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828848.834285:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828848.834293:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5485 00000100:00100000:0.0:1787828848.834296:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790574464/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828848.834298:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828848.834306:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787828848.834309:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787828848.834312:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787828848.834314:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828848.834318:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790574464:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828848.834326:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790575104/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828848.834327:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790574592:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787828848.834341:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828848.834344:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828848.834346:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787828848.834349:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828848.834354:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5485 00000100:00080000:2.0:1787828848.834358:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828848.834362:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828848/real 1787828848] req@ffff9e6804422d00 x1874668790574592/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828848.834364:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828848.834371:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828848.834378:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790574592/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828848.834384:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790575232/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828848.834388:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828848.834392:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828848.834394:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790574592:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828848.834406:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790574720:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828848.834422:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828848.834434:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828848/real 1787828848] req@ffff9e6804421680 x1874668790574720/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828848.834454:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790574720/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828848.834464:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828848.834468:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828848.834472:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790574720:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828848.834478:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790574848:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828848.834496:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828848.834504:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828848/real 1787828848] req@ffff9e6804423480 x1874668790574848/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828848.834512:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790574848/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828848.834519:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828848.834521:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828848.834523:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790574848:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828848.834527:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790574976:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828848.834537:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828848.834543:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828848/real 1787828848] req@ffff9e68044221c0 x1874668790574976/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828848.834550:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790574976/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828848.834560:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828848.834562:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828848.834564:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790574976:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828848.834567:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790575104:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828848.834577:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828848.834583:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828848/real 1787828848] req@ffff9e6804423840 x1874668790575104/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828848.834589:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790575104/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828848.834595:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828848.834597:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828848.834599:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790575104:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828848.834609:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790575232:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828848.834619:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828848.834625:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828848/real 1787828848] req@ffff9e6804422580 x1874668790575232/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828848.834632:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790575232/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828848.834637:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828848.834639:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828848.834642:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790575232:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828849.345891:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828849.537903:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828850.369782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828850.561870:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828851.393855:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828851.585893:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828852.417820:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828852.609891:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828853.441871:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828853.633894:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828853.953779:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828853.953787:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828853.953791:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828853.953800:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5490 00000100:00080000:3.0:1787828853.953809:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828853.953814:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828853.953844:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790575360/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828853.953865:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828853.953868:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828853.953870:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828853.953873:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5490 00000100:00080000:3.0:1787828853.953877:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828853.953880:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828853.953894:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828853.953900:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790575360:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828853.953909:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790575488/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828853.953920:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828853.953922:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828853.953924:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828853.953927:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5490 00000100:00080000:3.0:1787828853.953930:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828853.953933:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828853.953937:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828853.953945:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790575616/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828853.953952:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828853.953955:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828853.953956:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787828853.953960:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828853.953964:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5490 00000100:00080000:3.0:1787828853.953969:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828853.953972:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828853.953976:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828853.953984:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790575744/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828853.954005:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828853.954007:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828853.954008:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828853.954011:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5490 00000100:00080000:3.0:1787828853.954014:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828853.954019:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828853.954022:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828853.954024:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828853/real 1787828853] req@ffff9e680a2d52c0 x1874668790575360/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828853.954030:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790575872/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828853.954060:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828853.954063:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828853.954065:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828853.954070:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5490 00000100:00100000:0.0:1787828853.954074:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790575360/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828853.954076:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828853.954087:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787828853.954089:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828853.954094:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787828853.954095:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828853.954102:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790575360:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828853.954114:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790576000/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828853.954125:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790575488:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828853.954128:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828853.954132:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828853.954133:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828853.954138:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5490 00000100:00080000:3.0:1787828853.954143:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828853.954146:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828853.954150:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828853.954159:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828853.954162:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790576128/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828853.954171:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828853.954175:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828853.954178:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828853.954182:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828853/real 1787828853] req@ffff9e680a2d6580 x1874668790575488/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828853.954216:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5490 00000100:00100000:0.0:1787828853.954224:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790575488/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828853.954236:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828853.954239:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828853.954268:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828853.954269:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828853.954277:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790575488:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787828853.954285:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828853.954290:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790575616:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828853.954330:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790576256/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828853.954338:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828853.954363:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828853.954368:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828853.954371:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828853.954379:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828853/real 1787828853] req@ffff9e680a2d43c0 x1874668790575616/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828853.954379:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5490 00000100:00080000:3.0:1787828853.954388:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828853.954406:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828853.954412:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828853.954418:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790575616/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828853.954444:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790576384/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828853.954447:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828853.954459:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828853.954486:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790575616:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828853.954494:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790575744:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828853.954516:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828853.954526:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828853/real 1787828853] req@ffff9e680a2d5a40 x1874668790575744/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828853.954552:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790575744/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828853.954558:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828853.954560:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828853.954562:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790575744:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828853.954566:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790575872:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828853.954590:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828853.954597:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828853/real 1787828853] req@ffff9e680a2d4780 x1874668790575872/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828853.954604:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790575872/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828853.954610:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828853.954612:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828853.954614:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790575872:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828853.954638:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790576000:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828853.954658:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828853.954665:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828853/real 1787828853] req@ffff9e680a2d4000 x1874668790576000/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828853.954685:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790576000/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828853.954717:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828853.954719:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828853.954722:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790576000:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828853.954729:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790576128:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828853.954768:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828853.954781:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828853/real 1787828853] req@ffff9e680a2d52c0 x1874668790576128/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828853.954795:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790576128/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828853.954803:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828853.954806:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828853.954810:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790576128:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828853.954817:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790576256:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828853.954838:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828853.954850:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828853/real 1787828853] req@ffff9e680a2d70c0 x1874668790576256/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828853.954858:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790576256/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828853.954864:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828853.954866:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828853.954868:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790576256:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828853.954872:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790576384:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828853.954885:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828853.954891:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828853/real 1787828853] req@ffff9e680a2d7c00 x1874668790576384/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828853.954898:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790576384/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828878 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828853.954903:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828853.954905:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828853.954915:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790576384:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828854.465814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828854.657826:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828855.489861:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828855.681962:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828856.513767:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828856.705977:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828857.537804:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828857.730828:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828858.561925:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828858.753844:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828859.073748:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828859.073757:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828859.073761:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828859.073770:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5495 00000100:00080000:3.0:1787828859.073779:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828859.073786:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828859.073817:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790576512/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828859.073840:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828859.073843:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828859.073844:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828859.073847:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5495 00000100:00080000:3.0:1787828859.073850:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828859.073865:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828859.073873:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828859.073879:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790576512:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828859.073886:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790576640/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828859.073896:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828859.073898:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828859.073900:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828859.073903:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5495 00000100:00080000:3.0:1787828859.073906:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828859.073909:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828859.073912:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828859.073921:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790576768/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828859.073928:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828859.073930:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828859.073932:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828859.073934:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5495 00000100:00080000:3.0:1787828859.073938:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828859.073943:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828859.073946:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828859.073952:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828859.073962:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790576896/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828859.073988:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828859.073990:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828859.073992:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828859.073996:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828859/real 1787828859] req@ffff9e680a2d4f00 x1874668790576512/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828859.073997:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5495 00000100:00080000:3.0:1787828859.074003:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828859.074011:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828859.074017:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828859.074023:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790576512/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828859.074029:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790577024/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828859.074040:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828859.074042:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828859.074048:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828859.074048:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828859.074053:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828859.074058:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790576512:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828859.074061:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5495 00000100:00080000:3.0:1787828859.074067:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828859.074071:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828859.074084:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828859.074090:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790577152/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828859.074097:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790576640:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828859.074100:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828859.074103:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828859.074105:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828859.074109:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5495 00000100:00080000:3.0:1787828859.074114:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828859.074117:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828859.074121:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828859.074122:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828859.074134:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790577280/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828859.074138:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828859/real 1787828859] req@ffff9e680a2d61c0 x1874668790576640/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828859.074181:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828859.074185:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790576640/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828859.074186:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828859.074190:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828859.074198:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5495 00000100:00080000:0.0:1787828859.074204:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828859.074204:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828859.074213:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787828859.074214:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828859.074221:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790576640:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787828859.074221:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828859.074232:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790576768:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828859.074234:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790577408/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828859.074245:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828859.074248:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828859.074250:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787828859.074253:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828859.074258:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5495 00000100:00080000:3.0:1787828859.074263:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828859.074267:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828859/real 1787828859] req@ffff9e680a2d6d00 x1874668790576768/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828859.074278:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828859.074285:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790576768/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828859.074288:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828859.074302:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790577536/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828859.074303:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828859.074311:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828859.074319:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790576768:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828859.074328:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790576896:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828859.074344:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828859.074351:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828859/real 1787828859] req@ffff9e680a2d5e00 x1874668790576896/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828859.074358:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790576896/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828859.074365:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828859.074368:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828859.074372:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790576896:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828859.074378:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790577024:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828859.074411:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828859.074422:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828859/real 1787828859] req@ffff9e680a2d7480 x1874668790577024/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828859.074434:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790577024/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828859.074442:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828859.074446:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828859.074450:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790577024:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828859.074471:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790577152:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828859.074491:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828859.074508:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828859/real 1787828859] req@ffff9e680a2d4b40 x1874668790577152/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828859.074521:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790577152/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828859.074530:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828859.074533:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828859.074537:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790577152:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828859.074542:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790577280:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828859.074571:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828859.074584:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828859/real 1787828859] req@ffff9e680a2d4f00 x1874668790577280/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828859.074598:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790577280/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828859.074604:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828859.074606:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828859.074613:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790577280:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828859.074618:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790577408:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828859.074634:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828859.074660:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828859/real 1787828859] req@ffff9e680a2d7c00 x1874668790577408/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828859.074680:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790577408/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828859.074685:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828859.074687:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828859.074689:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790577408:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828859.074711:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790577536:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828859.074725:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828859.074740:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828859/real 1787828859] req@ffff9e680a2d70c0 x1874668790577536/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828859.074748:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790577536/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828859.074754:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828859.074756:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828859.074758:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790577536:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828859.585797:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828859.777943:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828860.609792:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828860.801945:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828861.633784:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828861.826795:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828862.657814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828862.849889:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828863.681814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828863.873853:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828864.193788:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828864.193798:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828864.193801:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828864.193811:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5500 00000100:00080000:3.0:1787828864.193822:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828864.193829:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828864.193862:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790577664/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828864.193885:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828864.193888:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828864.193889:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828864.193893:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5500 00000100:00080000:3.0:1787828864.193895:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828864.193898:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828864.193906:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828864.193916:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790577792/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828864.193925:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828864.193928:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828864.193929:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828864.193933:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5500 00000100:00080000:3.0:1787828864.193937:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828864.193939:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790577664:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828864.193941:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828864.193952:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828864.193969:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790577920/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828864.193982:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828864.193986:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828864.193988:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828864.193993:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5500 00000100:00080000:3.0:1787828864.193998:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828864.194004:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828864.194011:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828864.194019:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828864.194027:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790578048/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828864.194038:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828864.194041:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828864.194043:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828864.194056:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5500 00000100:00080000:3.0:1787828864.194063:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828864.194067:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828864.194071:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828864.194080:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828864/real 1787828864] req@ffff9e680a2d52c0 x1874668790577664/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828864.194084:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790578176/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828864.194111:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828864.194116:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828864.194119:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828864.194146:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5500 00000100:00080000:3.0:1787828864.194150:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828864.194154:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828864.194158:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828864.194164:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790577664/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828864.194171:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790578304/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828864.194208:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828864.194210:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828864.194219:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828864.194221:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828864.194224:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828864.194228:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5500 00000100:00080000:3.0:1787828864.194235:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828864.194241:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790577664:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828864.194241:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828864.194249:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828864.194274:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790578432/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828864.194281:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828864.194285:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828864.194287:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828864.194290:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5500 00000100:00080000:3.0:1787828864.194293:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828864.194296:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790577792:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828864.194299:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828864.194307:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828864.194320:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790578560/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828864.194329:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828864.194332:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828864.194334:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828864.194338:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5500 00000100:00080000:3.0:1787828864.194342:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828864.194349:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828864.194350:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828864.194358:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828864.194369:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790578688/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828864.194381:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828864/real 1787828864] req@ffff9e680a2d4000 x1874668790577792/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828864.194404:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790577792/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828864.194417:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828864.194433:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828864.194438:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790577792:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828864.194448:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790577920:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828864.194475:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828864.194492:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828864/real 1787828864] req@ffff9e680a2d4780 x1874668790577920/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828864.194509:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790577920/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828864.194520:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828864.194525:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828864.194529:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790577920:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828864.194538:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790578048:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828864.194566:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828864.194582:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828864/real 1787828864] req@ffff9e680a2d5a40 x1874668790578048/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828864.194599:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790578048/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828864.194657:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828864.194662:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828864.194666:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790578048:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828864.194674:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790578176:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828864.194724:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828864.194739:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828864/real 1787828864] req@ffff9e680a2d43c0 x1874668790578176/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828864.194754:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790578176/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828864.194764:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828864.194768:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828864.194772:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790578176:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828864.194780:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790578304:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828864.194805:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828864.194819:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828864/real 1787828864] req@ffff9e680a2d6580 x1874668790578304/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828864.194834:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790578304/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828864.194846:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828864.194849:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828864.194853:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790578304:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828864.194861:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790578432:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828864.194882:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828864.194895:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828864/real 1787828864] req@ffff9e680a2d70c0 x1874668790578432/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828864.194910:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790578432/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828864.194920:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828864.194925:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828864.194929:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790578432:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828864.194956:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790578560:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828864.194980:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828864.194994:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828864/real 1787828864] req@ffff9e680a2d7c00 x1874668790578560/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828864.195009:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790578560/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828864.195027:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828864.195031:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828864.195035:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790578560:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828864.195042:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790578688:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828864.195065:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828864.195078:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828864/real 1787828864] req@ffff9e680a2d4f00 x1874668790578688/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828864.195092:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790578688/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828864.195102:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828864.195105:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828864.195109:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790578688:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828864.705794:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828864.897866:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828865.729841:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828865.921864:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828866.753801:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828866.945883:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828867.777836:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828867.969874:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828868.801843:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828868.993945:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828869.313766:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828869.313777:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828869.313782:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828869.313796:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5506 00000100:00080000:3.0:1787828869.313810:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828869.313816:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828869.313866:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790578816/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828869.313894:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828869.313897:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828869.313899:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828869.313902:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5506 00000100:00080000:3.0:1787828869.313904:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828869.313907:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828869.313916:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828869.313926:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790578944/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828869.313939:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828869.313942:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828869.313944:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828869.313946:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5506 00000100:00080000:3.0:1787828869.313949:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828869.313952:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790578816:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828869.313953:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828869.313959:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828869.313967:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790579072/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828869.313975:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828869.313977:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828869.313978:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828869.313981:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5506 00000100:00080000:3.0:1787828869.313983:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828869.313985:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828869.313989:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828869.313997:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790579200/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828869.314003:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828869.314005:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828869.314006:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828869.314009:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5506 00000100:00080000:3.0:1787828869.314012:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828869.314014:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828869.314018:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828869.314025:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790579328/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828869.314030:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828869.314033:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828869.314035:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828869.314037:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5506 00000100:00080000:3.0:1787828869.314040:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828869.314042:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828869.314045:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828869.314048:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828869.314057:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790579456/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828869.314064:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828869.314067:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828869.314068:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828869.314072:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5506 00000100:00080000:3.0:1787828869.314074:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828869.314076:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828869.314080:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828869.314088:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790579584/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828869.314098:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828869/real 1787828869] req@ffff9e680a2d4b40 x1874668790578816/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828869.314101:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828869.314108:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828869.314112:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828869.314118:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5506 00000100:00080000:3.0:1787828869.314122:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828869.314125:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828869.314129:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790578816/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828869.314129:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828869.314143:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790579712/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828869.314144:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828869.314153:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828869.314158:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828869.314162:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790578816:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828869.314163:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828869.314167:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828869.314172:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5506 00000100:00080000:3.0:1787828869.314174:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828869.314177:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828869.314180:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828869.314186:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790579840/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828869.314197:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790578944:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828869.314217:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828869.314226:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828869/real 1787828869] req@ffff9e680a2d7480 x1874668790578944/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828869.314234:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790578944/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828869.314239:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828869.314241:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828869.314243:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790578944:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828869.314247:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790579072:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828869.314257:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828869.314263:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828869/real 1787828869] req@ffff9e680a2d5e00 x1874668790579072/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828869.314270:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790579072/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828869.314275:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828869.314277:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828869.314279:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790579072:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828869.314282:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790579200:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828869.314292:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828869.314298:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828869/real 1787828869] req@ffff9e680a2d6d00 x1874668790579200/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828869.314306:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790579200/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828869.314311:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828869.314313:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828869.314315:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790579200:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828869.314318:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790579328:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828869.314327:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828869.314333:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828869/real 1787828869] req@ffff9e680a2d61c0 x1874668790579328/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828869.314339:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790579328/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828869.314344:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828869.314346:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828869.314348:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790579328:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828869.314352:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790579456:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828869.314361:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828869.314367:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828869/real 1787828869] req@ffff9e680a2d4f00 x1874668790579456/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828869.314374:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790579456/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828869.314385:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828869.314387:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828869.314389:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790579456:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828869.314392:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790579584:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828869.314401:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828869.314407:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828869/real 1787828869] req@ffff9e680a2d7c00 x1874668790579584/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828869.314414:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790579584/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828869.314420:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828869.314421:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828869.314423:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790579584:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828869.314426:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790579712:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828869.314435:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828869.314441:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828869/real 1787828869] req@ffff9e680a2d70c0 x1874668790579712/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828869.314447:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790579712/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828869.314453:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828869.314454:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828869.314457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790579712:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828869.314473:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790579840:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828869.314483:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828869.314489:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828869/real 1787828869] req@ffff9e680a2d6580 x1874668790579840/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828869.314497:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790579840/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828869.314502:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828869.314504:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828869.314505:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790579840:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828869.825775:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828870.017860:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828870.849816:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828871.041887:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828871.873830:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828872.065845:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828872.897837:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828873.089877:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828873.921780:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828874.113913:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828874.433767:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828874.433776:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828874.433781:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828874.433791:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5511 00000100:00080000:3.0:1787828874.433800:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828874.433807:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828874.433839:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790579968/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828874.433860:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828874.433864:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828874.433866:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828874.433870:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5511 00000100:00080000:3.0:1787828874.433874:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828874.433878:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828874.433886:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828874.433896:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790580096/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828874.433904:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828874.433907:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828874.433908:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828874.433910:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5511 00000100:00080000:3.0:1787828874.433913:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828874.433916:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828874.433918:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828874.433927:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790580224/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828874.433939:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828874.433942:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828874.433944:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828874.433947:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5511 00000100:00080000:3.0:1787828874.433949:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828874.433953:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828874.433957:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828874.433967:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790580352/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828874.433974:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828874.433977:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828874.433978:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828874.433996:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5511 00000100:00080000:3.0:1787828874.433999:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828874.434002:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828874.434006:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828874.434013:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790580480/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828874.434019:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828874.434021:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828874.434022:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828874.434025:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5511 00000100:00080000:3.0:1787828874.434028:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828874.434033:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828874.434035:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790579968:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787828874.434038:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787828874.434091:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828874.434123:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790580608/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828874.434132:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828874/real 1787828874] req@ffff9e680a2d43c0 x1874668790579968/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828874.434133:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828874.434139:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828874.434142:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828874.434149:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5511 00000100:00080000:3.0:1787828874.434156:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828874.434160:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828874.434170:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790579968/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828874.434184:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828874.434190:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828874.434195:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790579968:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828874.434200:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790580096:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828874.434220:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828874.434230:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828874/real 1787828874] req@ffff9e680a2d5a40 x1874668790580096/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828874.434240:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790580096/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828874.434248:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828874.434252:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828874.434256:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790580096:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787828874.434256:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828874.434269:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790580224:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828874.434274:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790580736/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828874.434298:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828874.434312:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828874/real 1787828874] req@ffff9e680a2d4780 x1874668790580224/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828874.434324:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790580224/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828874.434331:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828874.434335:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828874.434335:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828874.434339:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828874.434343:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828874.434347:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5511 00000100:00100000:1.0:1787828874.434350:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790580224:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828874.434353:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828874.434359:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790580352:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787828874.434360:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828874.434369:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828874.434385:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790580864/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828874.434387:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828874.434398:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828874/real 1787828874] req@ffff9e680a2d4000 x1874668790580352/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828874.434423:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828874.434424:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790580352/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828874.434430:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828874.434433:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828874.434441:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828874.434442:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5511 00000100:00080000:3.0:1787828874.434448:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828874.434450:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828874.434454:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828874.434455:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790580352:192.168.203.109@tcp:8:kworker.0 00000008:00100000:3.0:1787828874.434460:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828874.434463:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790580480:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828874.434475:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790580992/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828874.434483:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828874.434495:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828874/real 1787828874] req@ffff9e680a2d52c0 x1874668790580480/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828874.434503:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790580480/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828874.434513:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828874.434515:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828874.434517:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790580480:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828874.434553:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790580608:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828874.434585:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828874.434593:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828874/real 1787828874] req@ffff9e680a2d6580 x1874668790580608/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828874.434601:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790580608/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828874.434608:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828874.434611:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828874.434613:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790580608:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828874.434617:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790580736:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828874.434640:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828874.434647:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828874/real 1787828874] req@ffff9e680a2d70c0 x1874668790580736/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828874.434655:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790580736/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828874.434660:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828874.434662:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828874.434665:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790580736:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828874.434669:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790580864:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828874.434680:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828874.434686:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828874/real 1787828874] req@ffff9e680a2d7c00 x1874668790580864/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828874.434693:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790580864/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828874.434725:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828874.434728:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828874.434731:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790580864:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828874.434738:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790580992:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828874.434752:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828874.434760:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828874/real 1787828874] req@ffff9e680a2d4f00 x1874668790580992/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828874.434768:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790580992/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828874.434773:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828874.434775:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828874.434778:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790580992:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828874.945776:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828875.137836:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828875.969777:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828876.161874:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828876.993781:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828877.185830:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828878.017789:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828878.209870:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828879.041826:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828879.233982:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828879.553857:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828879.553879:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828879.553884:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828879.553894:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5516 00000100:00080000:0.0:1787828879.553907:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828879.553914:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828879.553963:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790581120/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828879.553991:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828879.553994:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828879.553995:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828879.553999:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5516 00000100:00080000:0.0:1787828879.554001:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828879.554005:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828879.554016:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828879.554026:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790581248/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828879.554034:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828879.554037:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828879.554040:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828879.554041:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790581120:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828879.554045:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5516 00000100:00080000:0.0:1787828879.554050:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828879.554054:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828879.554057:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828879.554065:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790581376/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828879.554072:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828879.554075:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828879.554077:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828879.554079:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5516 00000100:00080000:0.0:1787828879.554091:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828879.554093:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828879.554097:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828879.554111:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790581504/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828879.554121:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828879.554126:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828879.554128:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828879.554132:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5516 00000800:00000100:1.0:1787828879.554139:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828879.554140:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828879.554148:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828879.554154:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828879.554181:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790581632/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828879.554192:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828879.554196:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828879.554198:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828879.554203:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5516 00000100:00100000:1.0:1787828879.554206:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828879/real 1787828879] req@ffff9e6803e9b480 x1874668790581120/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828879.554210:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828879.554221:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828879.554228:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828879.554230:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790581120/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828879.554243:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828879.554248:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790581760/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828879.554249:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828879.554255:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790581120:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828879.554271:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828879.554274:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828879.554276:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828879.554280:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5516 00000100:00080000:0.0:1787828879.554284:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828879.554288:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790581248:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828879.554290:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828879.554301:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828879.554318:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790581888/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828879.554328:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828879.554339:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828879.554342:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828879.554344:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828879.554349:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5516 00000100:00080000:0.0:1787828879.554352:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828879.554358:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828879.554358:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828879/real 1787828879] req@ffff9e6803e9a580 x1874668790581248/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828879.554366:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828879.554389:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790581248/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828879.554393:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790582016/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828879.554411:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828879.554413:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828879.554417:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787828879.554420:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787828879.554423:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790581248:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828879.554424:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828879.554445:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5516 00000100:00100000:1.0:1787828879.554446:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790581376:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828879.554448:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828879.554453:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828879.554457:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828879.554465:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790582144/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828879.554471:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828879.554484:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828879/real 1787828879] req@ffff9e6803e99680 x1874668790581376/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828879.554518:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790581376/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828879.554524:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828879.554526:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828879.554529:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790581376:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828879.554534:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790581504:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828879.554565:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828879.554577:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828879/real 1787828879] req@ffff9e6803e99e00 x1874668790581504/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828879.554591:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790581504/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828879.554600:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828879.554603:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828879.554605:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790581504:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828879.554610:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790581632:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828879.554625:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828879.554633:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828879/real 1787828879] req@ffff9e6803e9b840 x1874668790581632/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828879.554649:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790581632/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828879.554660:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828879.554662:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828879.554665:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790581632:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828879.554668:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790581760:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828879.554678:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828879.554684:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828879/real 1787828879] req@ffff9e6803e9a1c0 x1874668790581760/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828879.554691:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790581760/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828879.554728:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828879.554732:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828879.554736:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790581760:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828879.554758:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790581888:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828879.554777:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828879.554786:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828879/real 1787828879] req@ffff9e6803e98780 x1874668790581888/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828879.554794:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790581888/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828879.554799:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828879.554811:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828879.554814:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790581888:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828879.554818:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790582016:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828879.554829:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828879.554834:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828879/real 1787828879] req@ffff9e6803e98b40 x1874668790582016/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828879.554842:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790582016/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828879.554847:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828879.554849:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828879.554851:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790582016:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828879.554854:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790582144:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828879.554863:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828879.554878:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828879/real 1787828879] req@ffff9e6803e9bc00 x1874668790582144/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828879.554885:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790582144/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828879.554890:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828879.554892:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828879.554893:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790582144:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828880.065768:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828880.257889:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828881.089782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828881.281859:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828882.113789:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:0.0:1787828882.305921:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828883.137843:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828883.329859:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828884.161842:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828884.353947:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828884.673861:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828884.673874:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828884.673879:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828884.673891:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5521 00000100:00080000:0.0:1787828884.673904:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828884.673912:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828884.673955:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790582272/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828884.674021:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828884.674028:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828884.674031:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828884.674037:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5521 00000100:00080000:0.0:1787828884.674042:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828884.674065:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828884.674076:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828884.674089:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790582272:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787828884.674099:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790582400/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828884.674139:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828884.674145:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828884.674148:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828884.674156:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5521 00000100:00080000:0.0:1787828884.674163:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828884.674170:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828884.674195:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828884.674219:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790582528/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828884.674238:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828884.674267:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828884.674271:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828884.674274:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828884.674282:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5521 00000100:00080000:0.0:1787828884.674290:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828884.674296:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828884.674304:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828884.674338:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790582656/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828884.674349:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828884.674355:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787828884.674357:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828884/real 1787828884] req@ffff9e6803e992c0 x1874668790582272/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828884.674359:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828884.674371:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5521 00000100:00080000:0.0:1787828884.674396:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828884.674402:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828884.674412:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790582272/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828884.674417:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828884.674439:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828884.674447:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790582784/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828884.674474:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828884.674480:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790582272:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787828884.674502:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828884.674505:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828884.674508:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828884.674513:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790582400:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828884.674537:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5521 00000100:00080000:0.0:1787828884.674545:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828884.674553:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1787828884.674561:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787828884.674562:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828884.674578:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828884/real 1787828884] req@ffff9e6803e98f00 x1874668790582400/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828884.674587:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790582912/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828884.674609:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790582400/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828884.674618:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828884.674624:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828884.674625:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828884.674639:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828884.674642:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787828884.674654:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5521 00000100:00100000:1.0:1787828884.674656:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790582400:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828884.674664:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828884.674674:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828884.674676:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790582528:192.168.203.109@tcp:38:kworker.0 00000008:00100000:0.0:1787828884.674686:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828884.674705:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790583040/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828884.674733:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828884.674754:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828884/real 1787828884] req@ffff9e6803e9ad00 x1874668790582528/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828884.674769:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790582528/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828884.674841:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828884.674849:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787828884.674857:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828884.674859:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790582528:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787828884.674878:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828884.674883:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828884.674884:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790582656:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787828884.674896:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5521 00000100:00080000:0.0:1787828884.674906:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828884.674912:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1787828884.674920:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787828884.674946:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828884.674947:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828884/real 1787828884] req@ffff9e6803e9a940 x1874668790582656/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828884.674970:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790582656/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828884.674981:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787828884.674985:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790583168/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828884.674990:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828884.675007:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790582656:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828884.675037:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790582784:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787828884.675062:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828884.675067:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787828884.675074:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787828884.675075:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828884.675085:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5521 00000100:00080000:0.0:1787828884.675093:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828884.675099:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828884/real 1787828884] req@ffff9e6803e9bc00 x1874668790582784/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828884.675121:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787828884.675138:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828884.675138:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790582784/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828884.675166:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828884.675179:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828884.675183:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790582784:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828884.675191:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790583296/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828884.675204:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790582912:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828884.675238:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828884.675263:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828884/real 1787828884] req@ffff9e6803e98b40 x1874668790582912/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828884.675289:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790582912/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828884.675300:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828884.675304:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828884.675308:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790582912:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828884.675314:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790583040:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828884.675342:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828884.675350:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828884/real 1787828884] req@ffff9e6803e98780 x1874668790583040/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828884.675358:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790583040/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828884.675363:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828884.675366:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828884.675368:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790583040:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828884.675373:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790583168:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828884.675384:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828884.675390:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828884/real 1787828884] req@ffff9e6803e9a1c0 x1874668790583168/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828884.675397:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790583168/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828884.675402:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828884.675416:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828884.675417:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790583168:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828884.675427:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790583296:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828884.675438:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828884.675445:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828884/real 1787828884] req@ffff9e6803e9b840 x1874668790583296/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828884.675453:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790583296/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828884.675473:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828884.675484:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828884.675486:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790583296:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828885.185829:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828885.377905:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828886.209849:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828886.401929:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828887.233775:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828887.425888:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828888.257841:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828888.449949:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828889.281853:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828889.474021:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828889.793855:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828889.793879:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828889.793884:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828889.793899:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5526 00000100:00080000:2.0:1787828889.793914:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828889.793924:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828889.793971:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790583424/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828889.794000:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828889.794003:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828889.794004:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828889.794008:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5526 00000100:00080000:2.0:1787828889.794013:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828889.794017:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828889.794027:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828889.794040:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790583552/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828889.794047:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790583424:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828889.794052:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828889.794063:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828889.794065:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828889.794068:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5526 00000100:00080000:2.0:1787828889.794070:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828889.794074:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828889.794078:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828889.794089:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790583680/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828889.794101:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828889.794105:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828889.794106:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828889.794110:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5526 00000100:00080000:2.0:1787828889.794112:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828889.794114:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828889.794118:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828889.794134:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828889.794135:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790583808/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828889.794149:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828889.794154:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828889.794157:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828889.794163:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5526 00000100:00080000:2.0:1787828889.794168:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828889.794174:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828889.794179:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828889.794183:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828889/real 1787828889] req@ffff9e6804420780 x1874668790583424/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828889.794214:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790583424/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828889.794217:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790583936/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828889.794273:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828889.794282:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828889.794287:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790583424:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828889.794306:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828889.794311:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828889.794314:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828889.794318:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5526 00000100:00080000:2.0:1787828889.794324:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828889.794328:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790583552:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828889.794335:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828889.794342:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828889.794357:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790584064/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828889.794366:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828889.794384:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828889.794389:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828889/real 1787828889] req@ffff9e68044212c0 x1874668790583552/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828889.794390:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828889.794395:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828889.794409:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5526 00000100:00080000:2.0:1787828889.794420:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828889.794441:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828889.794442:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790583552/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828889.794450:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787828889.794473:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828889.794478:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828889.794484:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790583552:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828889.794491:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790584192/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828889.794500:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790583680:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828889.794527:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828889.794536:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828889.794538:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828889.794550:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5526 00000100:00080000:2.0:1787828889.794556:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828889.794561:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828889.794570:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828889.794576:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828889.794601:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828889/real 1787828889] req@ffff9e68044230c0 x1874668790583680/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828889.794620:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790583680/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828889.794620:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790584320/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828889.794685:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828889.794703:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787828889.794709:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828889.794716:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787828889.794730:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790583680:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828889.794741:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790583808:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828889.794771:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828889.794772:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828889.794783:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5526 00000100:00080000:2.0:1787828889.794789:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828889.794795:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828889.794802:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828889.794803:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828889/real 1787828889] req@ffff9e6804420b40 x1874668790583808/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828889.794826:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790584448/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828889.794830:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790583808/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828889.794865:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828889.794877:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828889.794882:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790583808:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828889.794890:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790583936:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828889.794909:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828889.794924:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828889/real 1787828889] req@ffff9e6804422580 x1874668790583936/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828889.794938:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790583936/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828889.794951:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828889.794955:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828889.794959:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790583936:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828889.794980:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790584064:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828889.794997:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828889.795004:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828889/real 1787828889] req@ffff9e6804423840 x1874668790584064/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828889.795012:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790584064/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828889.795019:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828889.795021:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828889.795024:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790584064:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828889.795027:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790584192:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828889.795038:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828889.795043:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828889/real 1787828889] req@ffff9e68044221c0 x1874668790584192/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828889.795051:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790584192/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828889.795065:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828889.795068:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828889.795072:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790584192:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828889.795078:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790584320:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828889.795099:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828889.795111:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828889/real 1787828889] req@ffff9e6804423480 x1874668790584320/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828889.795122:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790584320/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828889.795128:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828889.795130:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828889.795132:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790584320:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828889.795137:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790584448:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828889.795151:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828889.795158:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828889/real 1787828889] req@ffff9e6804421680 x1874668790584448/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828889.795165:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790584448/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828889.795170:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828889.795172:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828889.795174:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790584448:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787828890.305839:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828890.497922:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828891.329817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828891.521902:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828892.353838:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828892.545925:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828893.377784:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828893.569838:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828894.401850:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828894.593897:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828894.913876:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828894.913895:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828894.913903:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828894.913918:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5531 00000100:00080000:3.0:1787828894.913946:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828894.913958:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828894.914028:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790584576/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828894.914052:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828894.914056:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828894.914057:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828894.914060:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5531 00000100:00080000:3.0:1787828894.914063:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828894.914076:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828894.914084:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828894.914098:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790584704/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828894.914107:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828894.914110:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828894.914113:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828894.914116:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5531 00000100:00080000:3.0:1787828894.914118:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828894.914120:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828894.914123:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828894.914135:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790584832/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828894.914137:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790584576:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828894.914145:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828894.914173:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828894.914176:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828894.914193:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5531 00000100:00080000:3.0:1787828894.914201:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828894.914206:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828894.914212:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828894.914243:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790584960/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828894.914251:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828894.914253:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828894.914254:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828894.914256:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5531 00000100:00080000:3.0:1787828894.914260:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828894.914264:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828894.914268:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828894.914275:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790585088/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828894.914286:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828894.914324:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828894.914329:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828894.914331:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828894.914336:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5531 00000100:00080000:3.0:1787828894.914341:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828894.914346:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828894.914355:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828894.914372:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790585216/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828894.914383:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828894/real 1787828894] req@ffff9e680a2d61c0 x1874668790584576/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828894.914405:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828894.914428:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787828894.914431:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790584576/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828894.914431:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828894.914446:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5531 00000100:00080000:3.0:1787828894.914460:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828894.914464:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828894.914465:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787828894.914477:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:3.0:1787828894.914498:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828894.914500:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790584576:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828894.914517:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790584704:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828894.914525:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790585344/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828894.914541:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828894.914545:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828894.914549:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828894.914555:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5531 00000800:00000100:1.0:1787828894.914564:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828894.914565:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828894.914578:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828894.914584:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828894.914616:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828894/real 1787828894] req@ffff9e680a2d6d00 x1874668790584704/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828894.914642:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790585472/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828894.914651:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828894.914653:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790584704/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828894.914657:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828894.914665:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828894.914683:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828894.914688:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5531 00000100:00080000:1.0:1787828894.914733:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828894.914738:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828894.914743:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790584704:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828894.914751:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828894.914760:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828894.914774:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790585600/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828894.914799:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790584832:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828894.914847:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828894.914863:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828894/real 1787828894] req@ffff9e680a2d5e00 x1874668790584832/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828894.914888:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790584832/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828894.914898:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828894.914902:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828894.914905:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790584832:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828894.914911:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790584960:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828894.914925:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828894.914932:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828894/real 1787828894] req@ffff9e680a2d7480 x1874668790584960/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828894.914941:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790584960/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828894.914948:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828894.914950:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828894.914953:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790584960:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828894.914957:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790585088:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828894.914978:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828894.914991:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828894/real 1787828894] req@ffff9e680a2d4b40 x1874668790585088/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828894.915030:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790585088/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828894.915045:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828894.915047:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828894.915050:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790585088:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828894.915055:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790585216:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828894.915091:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828894.915101:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828894/real 1787828894] req@ffff9e680a2d4f00 x1874668790585216/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828894.915113:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790585216/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828894.915125:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828894.915129:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828894.915133:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790585216:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828894.915142:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790585344:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828894.915169:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828894.915180:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828894/real 1787828894] req@ffff9e680a2d7c00 x1874668790585344/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828894.915190:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790585344/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828894.915196:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828894.915199:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828894.915202:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790585344:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828894.915207:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790585472:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828894.915223:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828894.915230:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828894/real 1787828894] req@ffff9e680a2d70c0 x1874668790585472/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828894.915239:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790585472/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828894.915245:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828894.915247:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828894.915249:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790585472:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828894.915279:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790585600:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828894.915309:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828894.915316:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828894/real 1787828894] req@ffff9e680a2d6580 x1874668790585600/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828894.915324:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790585600/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828919 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828894.915329:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828894.915332:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828894.915334:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790585600:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828895.425783:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828895.617986:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828896.449821:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828896.641914:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828897.473771:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828897.666027:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828898.497849:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828898.689808:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828899.521770:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828899.713897:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828900.033869:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828900.033880:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828900.033885:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828900.033896:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5536 00000100:00080000:2.0:1787828900.033909:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828900.033916:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828900.033968:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790585728/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828900.034020:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828900.034024:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828900.034026:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828900.034030:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5536 00000100:00080000:2.0:1787828900.034036:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828900.034042:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828900.034052:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828900.034069:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790585728:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787828900.034069:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790585856/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828900.034089:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828900.034092:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828900.034094:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828900.034099:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5536 00000100:00080000:2.0:1787828900.034102:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828900.034104:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828900.034108:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828900.034119:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790585984/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828900.034127:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828900.034130:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828900.034131:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828900.034134:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5536 00000100:00080000:2.0:1787828900.034138:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828900.034147:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828900.034151:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828900.034163:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790586112/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828900.034177:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828900.034180:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828900.034183:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828900.034188:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5536 00000100:00080000:2.0:1787828900.034191:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828900.034195:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828900.034199:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828900.034203:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790585856:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828900.034214:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790586240/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828900.034226:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828900.034229:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828900.034231:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828900.034235:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5536 00000100:00080000:2.0:1787828900.034241:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828900.034246:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790585984:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828900.034253:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828900.034262:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828900.034285:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790586368/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828900.034314:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790586112:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787828900.034314:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828900.034333:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828900.034337:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828900.034343:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5536 00000100:00080000:2.0:1787828900.034349:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828900.034367:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828900.034373:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828900.034386:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790586496/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828900.034393:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790586240:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787828900.034398:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828900.034401:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828900.034403:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828900.034412:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5536 00000100:00080000:2.0:1787828900.034416:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828900.034422:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828900.034425:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828900.034435:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790586624/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828900.034442:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828900.034447:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828900.034449:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828900.034453:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790586368:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828900.034457:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5536 00000100:00080000:2.0:1787828900.034471:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828900.034477:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828900.034482:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828900.034493:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790586496:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828900.034501:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790586752/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828900.034514:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790586624:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828900.034550:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790586752:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828900.545843:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828900.737866:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828901.569833:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828901.761914:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828902.593806:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828902.785855:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787828903.105975:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:2.0:1787828903.106015:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787828903.106030:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787828903.106080:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:2.0:1787828903.106103:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:1.0:1787828903.106111:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828900/real 1787828903] req@ffff9e6804422580 x1874668790586752/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787828903.106119:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787828903.106142:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:1.0:1787828903.106166:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790586752/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787828903.106167:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00080000:1.0:1787828903.106189:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000800:00000100:2.0:1787828903.106200:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00080000:1.0:1787828903.106207:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828903.106218:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790586752:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787828903.106231:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:1.0:1787828903.106244:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828900/real 1787828903] req@ffff9e6804421680 x1874668790586240/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787828903.106265:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:1.0:1787828903.106303:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790586240/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828903.106314:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828903.106320:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828903.106325:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790586240:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828903.106352:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828900/real 1787828903] req@ffff9e6804423480 x1874668790586368/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828903.106369:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790586368/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828903.106381:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828903.106385:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828903.106389:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790586368:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828903.106398:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828900/real 1787828903] req@ffff9e6804422940 x1874668790585856/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828903.106415:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790585856/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828903.106426:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828903.106431:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828903.106436:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790585856:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828903.106452:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828900/real 1787828903] req@ffff9e6804421e00 x1874668790585984/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828903.106470:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790585984/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828903.106480:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828903.106485:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828903.106489:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790585984:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828903.106498:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828900/real 1787828903] req@ffff9e6804420f00 x1874668790586112/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828903.106513:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790586112/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828903.106523:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828903.106528:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828903.106532:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790586112:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828903.106542:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828900/real 1787828903] req@ffff9e6804422d00 x1874668790585728/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828903.106559:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790585728/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828903.106569:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828903.106573:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828903.106577:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790585728:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828903.106624:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828900/real 1787828903] req@ffff9e68044221c0 x1874668790586496/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828903.106640:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790586496/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828903.106649:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828903.106653:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828903.106656:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790586496:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828903.106664:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828900/real 1787828903] req@ffff9e6804423840 x1874668790586624/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828903.106676:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790586624/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828903.106685:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828903.106691:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828903.106734:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790586624:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828903.617855:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828903.809855:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828904.641785:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828904.833850:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828905.153834:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828905.153847:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828905.153853:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828905.153866:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5541 00000100:00080000:3.0:1787828905.153880:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828905.153891:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828905.153930:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790586880/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828905.153962:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828905.153967:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828905.153969:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828905.153975:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5541 00000100:00080000:3.0:1787828905.153981:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828905.153986:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828905.153999:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828905.154017:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790587008/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828905.154033:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828905.154039:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787828905.154040:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790586880:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828905.154044:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828905.154057:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5541 00000100:00080000:3.0:1787828905.154063:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828905.154069:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828905.154077:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828905.154095:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790587136/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828905.154109:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828905.154114:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828905.154118:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828905.154124:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5541 00000100:00080000:3.0:1787828905.154129:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828905.154135:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1787828905.154143:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787828905.154145:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828905.154168:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790587264/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828905.154183:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828905.154188:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828905.154191:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828905.154199:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828905/real 1787828905] req@ffff9e680a2d52c0 x1874668790586880/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828905.154199:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5541 00000100:00080000:3.0:1787828905.154208:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828905.154219:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828905.154230:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828905.154241:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790586880/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828905.154251:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790587392/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828905.154266:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828905.154271:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828905.154277:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828905.154281:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828905.154287:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828905.154290:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790586880:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828905.154298:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5541 00000100:00080000:3.0:1787828905.154307:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828905.154313:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828905.154339:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828905.154350:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790587008:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828905.154362:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790587520/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828905.154379:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828905.154384:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828905.154386:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828905.154393:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828905.154397:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5541 00000100:00080000:3.0:1787828905.154406:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828905.154412:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828905.154420:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828905.154421:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828905/real 1787828905] req@ffff9e680a2d4000 x1874668790587008/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828905.154444:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790587648/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828905.154448:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790587008/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828905.154459:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828905.154463:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828905.154474:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790587008:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828905.154496:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828905.154500:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790587136:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828905.154502:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828905.154508:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828905.154518:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5541 00000100:00080000:3.0:1787828905.154527:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828905.154533:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1787828905.154542:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787828905.154542:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828905.154567:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828905/real 1787828905] req@ffff9e680a2d4780 x1874668790587136/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828905.154569:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790587776/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828905.154599:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790587136/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828905.154602:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828905.154615:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828905.154620:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828905.154625:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828905.154632:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5541 00000100:00080000:1.0:1787828905.154638:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828905.154642:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828905.154651:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790587136:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828905.154654:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828905.154668:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828905.154673:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790587264:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828905.154690:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790587904/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828905.154768:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828905.154790:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828905/real 1787828905] req@ffff9e680a2d5a40 x1874668790587264/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828905.154805:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790587264/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828905.154815:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828905.154820:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828905.154824:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790587264:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828905.154831:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790587392:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828905.154857:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828905.154871:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828905/real 1787828905] req@ffff9e680a2d43c0 x1874668790587392/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828905.154887:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790587392/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828905.154899:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828905.154903:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828905.154907:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790587392:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828905.154934:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790587520:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828905.154964:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828905.154981:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828905/real 1787828905] req@ffff9e680a2d52c0 x1874668790587520/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828905.154999:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790587520/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828905.155013:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828905.155017:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828905.155022:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790587520:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828905.155030:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790587648:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828905.155059:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828905.155075:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828905/real 1787828905] req@ffff9e680a2d6580 x1874668790587648/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828905.155092:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790587648/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828905.155103:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828905.155108:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828905.155112:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790587648:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828905.155121:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790587776:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828905.155149:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828905.155175:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828905/real 1787828905] req@ffff9e680a2d70c0 x1874668790587776/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828905.155191:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790587776/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828905.155203:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828905.155207:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828905.155212:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790587776:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828905.155828:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790587904:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828905.155858:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828905.155871:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828905/real 1787828905] req@ffff9e680a2d7c00 x1874668790587904/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828905.155883:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790587904/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828905.155890:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828905.155893:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828905.155896:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790587904:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828905.665798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828905.857880:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828906.689843:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828906.881922:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828907.713809:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828907.905888:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828908.737813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828908.929822:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828909.761780:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828909.953823:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828910.273768:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828910.273777:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828910.273781:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828910.273789:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5546 00000100:00080000:3.0:1787828910.273798:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828910.273804:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828910.273833:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790588032/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828910.273855:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828910.273857:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828910.273858:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828910.273861:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5546 00000100:00080000:3.0:1787828910.273864:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828910.273868:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828910.273876:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828910.273885:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790588160/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828910.273892:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790588032:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828910.273895:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828910.273900:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828910.273903:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828910.273906:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5546 00000100:00080000:3.0:1787828910.273908:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828910.273911:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828910.273914:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828910.273922:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790588288/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828910.273929:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828910.273931:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828910.273932:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828910.273935:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5546 00000100:00080000:3.0:1787828910.273937:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828910.273940:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828910.273943:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828910.273949:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790588416/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828910.273960:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787828910.273963:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828910.273964:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828910.273968:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828910.273971:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5546 00000100:00080000:3.0:1787828910.273973:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828910.273976:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828910.273979:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828910.273987:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790588544/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828910.273993:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828910.273995:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828910.273996:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828910.273999:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5546 00000100:00080000:3.0:1787828910.274001:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828910.274004:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828910.274007:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828910.274020:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828910/real 1787828910] req@ffff9e680a2d4f00 x1874668790588032/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828910.274025:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790588672/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828910.274039:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828910.274041:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828910.274042:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828910.274046:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790588032/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828910.274048:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5546 00000100:00080000:3.0:1787828910.274053:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828910.274059:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828910.274064:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787828910.274065:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828910.274071:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828910.274074:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790588800/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828910.274075:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790588032:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828910.274087:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828910.274089:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828910.274091:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828910.274105:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5546 00000100:00080000:3.0:1787828910.274106:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828910.274109:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828910.274112:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828910.274114:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790588160:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828910.274121:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790588928/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828910.274130:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828910.274132:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828910.274134:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828910.274137:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5546 00000800:00000100:1.0:1787828910.274141:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828910.274142:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828910.274147:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828910.274151:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828910.274155:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828910/real 1787828910] req@ffff9e680a2d4b40 x1874668790588160/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828910.274161:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790589056/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828910.274177:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790588160/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828910.274192:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828910.274195:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828910.274198:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790588160:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828910.274202:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790588288:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828910.274217:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828910.274223:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828910/real 1787828910] req@ffff9e680a2d7480 x1874668790588288/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828910.274230:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790588288/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828910.274235:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828910.274237:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828910.274239:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790588288:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828910.274243:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790588416:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828910.274253:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828910.274261:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828910/real 1787828910] req@ffff9e680a2d5e00 x1874668790588416/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828910.274302:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790588416/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828910.274319:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828910.274322:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828910.274325:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790588416:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828910.274330:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790588544:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828910.274352:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828910.274363:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828910/real 1787828910] req@ffff9e680a2d6d00 x1874668790588544/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828910.274370:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790588544/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828910.274376:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828910.274379:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828910.274381:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790588544:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828910.274385:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790588672:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828910.274398:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828910.274410:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828910/real 1787828910] req@ffff9e680a2d61c0 x1874668790588672/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828910.274420:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790588672/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828910.274432:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828910.274434:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828910.274437:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790588672:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828910.274441:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790588800:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828910.274453:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828910.274461:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828910/real 1787828910] req@ffff9e680a2d7c00 x1874668790588800/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828910.274469:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790588800/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828910.274476:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828910.274478:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828910.274481:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790588800:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828910.274514:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790588928:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828910.274530:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828910.274543:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828910/real 1787828910] req@ffff9e680a2d70c0 x1874668790588928/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828910.274557:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790588928/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828910.274566:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828910.274570:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828910.274575:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790588928:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828910.274583:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790589056:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828910.274608:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828910.274623:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828910/real 1787828910] req@ffff9e680a2d6580 x1874668790589056/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828910.274632:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790589056/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828910.274638:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828910.274640:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828910.274642:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790589056:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828910.785797:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828910.977826:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828911.809802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828912.001860:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828912.833794:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828913.025875:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828913.857833:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828914.049873:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828914.881852:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828915.073891:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828915.393784:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828915.393798:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828915.393804:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828915.393817:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5552 00000100:00080000:3.0:1787828915.393831:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828915.393840:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828915.393895:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790589184/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828915.393926:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828915.393930:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828915.393932:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828915.393937:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5552 00000100:00080000:3.0:1787828915.393948:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828915.393957:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828915.393969:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828915.393985:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790589312/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828915.393989:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790589184:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828915.394006:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828915.394010:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828915.394012:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828915.394017:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5552 00000100:00080000:3.0:1787828915.394022:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828915.394033:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828915.394042:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828915.394054:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790589440/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828915.394065:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828915.394069:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828915.394071:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828915.394077:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828915.394080:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5552 00000100:00080000:3.0:1787828915.394088:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828915.394093:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828915.394099:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828915.394114:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790589568/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828915.394125:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828915.394129:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828915.394131:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828915.394138:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828915/real 1787828915] req@ffff9e680a2d52c0 x1874668790589184/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828915.394139:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5552 00000100:00080000:3.0:1787828915.394147:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828915.394155:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828915.394162:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828915.394170:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790589184/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828915.394178:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790589696/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828915.394193:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828915.394194:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828915.394202:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828915.394204:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828915.394206:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828915.394211:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790589184:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828915.394216:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5552 00000100:00080000:3.0:1787828915.394222:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828915.394228:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828915.394233:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828915.394242:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790589312:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828915.394259:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790589824/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828915.394274:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828915.394279:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:1.0:1787828915.394283:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828915.394283:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828915.394292:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5552 00000100:00080000:3.0:1787828915.394297:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828915.394301:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828915/real 1787828915] req@ffff9e680a2d43c0 x1874668790589312/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828915.394304:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828915.394312:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828915.394318:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790589312/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828915.394326:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790589952/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828915.394328:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828915.394336:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828915.394341:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790589312:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828915.394352:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790589440:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828915.394361:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828915.394366:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828915.394369:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787828915.394372:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828915.394380:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5552 00000100:00100000:1.0:1787828915.394384:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828915/real 1787828915] req@ffff9e680a2d5a40 x1874668790589440/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828915.394387:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828915.394397:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828915.394399:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790589440/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828915.394403:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828915.394411:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828915.394415:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828915.394418:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790589440:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787828915.394422:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790590080/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828915.394427:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790589568:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787828915.394435:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828915.394439:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828915.394440:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828915.394444:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5552 00000100:00080000:3.0:1787828915.394448:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828915.394451:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828915.394455:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787828915.394460:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787828915.394466:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790590208/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828915.394476:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828915/real 1787828915] req@ffff9e680a2d4780 x1874668790589568/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828915.394497:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790589568/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828915.394506:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828915.394510:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828915.394513:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790589568:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828915.394527:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790589696:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828915.394550:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828915.394562:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828915/real 1787828915] req@ffff9e680a2d4000 x1874668790589696/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828915.394575:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790589696/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828915.394585:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828915.394588:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828915.394592:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790589696:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828915.394617:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790589824:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828915.394645:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828915.394658:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828915/real 1787828915] req@ffff9e680a2d52c0 x1874668790589824/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828915.394670:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790589824/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828915.394683:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828915.394685:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828915.394687:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790589824:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828915.394692:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790589952:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828915.394733:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828915.394741:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828915/real 1787828915] req@ffff9e680a2d6580 x1874668790589952/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828915.394748:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790589952/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828915.394754:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828915.394756:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828915.394758:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790589952:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828915.394762:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790590080:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828915.394780:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828915.394787:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828915/real 1787828915] req@ffff9e680a2d70c0 x1874668790590080/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828915.394794:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790590080/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828915.394799:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828915.394800:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828915.394803:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790590080:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828915.394807:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790590208:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828915.394818:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828915.394823:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828915/real 1787828915] req@ffff9e680a2d7c00 x1874668790590208/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828915.394830:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790590208/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828915.394835:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828915.394837:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828915.394839:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790590208:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828915.905835:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828916.097905:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828916.929822:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828917.121921:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828917.953804:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828918.145828:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828918.977800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828919.169888:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828920.001811:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828920.193817:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828920.513787:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828920.513795:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828920.513799:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828920.513807:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5557 00000100:00080000:3.0:1787828920.513816:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828920.513821:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828920.513854:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790590336/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828920.513876:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828920.513879:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828920.513880:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828920.513883:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5557 00000100:00080000:3.0:1787828920.513885:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828920.513888:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828920.513895:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828920.513904:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790590464/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828920.513912:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828920.513915:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828920.513916:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828920.513920:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5557 00000100:00100000:1.0:1787828920.513922:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790590336:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828920.513923:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828920.513929:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828920.513932:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828920.513939:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790590592/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828920.513945:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828920.513947:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828920.513948:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828920.513950:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5557 00000100:00080000:3.0:1787828920.513952:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828920.513954:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828920.513957:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828920.513964:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790590720/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828920.513969:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828920.513972:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828920.513973:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828920.513975:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5557 00000100:00080000:3.0:1787828920.513978:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828920.513980:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828920.513982:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828920.513989:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790590848/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828920.514000:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787828920.514002:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828920.514004:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828920.514007:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828920.514010:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5557 00000100:00080000:3.0:1787828920.514013:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828920.514016:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828920.514019:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828920.514027:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790590976/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828920.514032:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828920.514035:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828920.514036:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828920.514039:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5557 00000100:00080000:3.0:1787828920.514042:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828920.514044:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828920.514048:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828920.514052:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828920/real 1787828920] req@ffff9e680a2d61c0 x1874668790590336/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828920.514059:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790591104/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828920.514078:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828920.514080:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828920.514081:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828920.514084:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5557 00000100:00100000:1.0:1787828920.514089:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790590336/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828920.514089:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828920.514125:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828920.514135:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787828920.514140:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828920.514148:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787828920.514150:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790591232/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828920.514156:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790590336:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828920.514180:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828920.514184:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828920.514187:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828920.514194:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5557 00000100:00100000:1.0:1787828920.514196:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790590464:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828920.514199:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828920.514208:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828920.514249:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828920.514275:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790591360/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787828920.514278:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828920.514312:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828920/real 1787828920] req@ffff9e680a2d6d00 x1874668790590464/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828920.514329:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790590464/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828920.514338:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828920.514341:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828920.514344:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790590464:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828920.514350:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790590592:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828920.514367:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828920.514376:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828920/real 1787828920] req@ffff9e680a2d5e00 x1874668790590592/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828920.514386:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790590592/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828920.514405:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828920.514407:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828920.514410:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790590592:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828920.514415:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790590720:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828920.514429:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828920.514437:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828920/real 1787828920] req@ffff9e680a2d7480 x1874668790590720/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828920.514453:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790590720/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828920.514471:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828920.514473:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828920.514477:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790590720:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828920.514484:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790590848:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828920.514523:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828920.514534:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828920/real 1787828920] req@ffff9e680a2d4b40 x1874668790590848/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828920.514547:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790590848/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828920.514556:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828920.514560:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828920.514563:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790590848:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828920.514567:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790590976:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828920.514582:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828920.514590:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828920/real 1787828920] req@ffff9e680a2d4f00 x1874668790590976/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828920.514597:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790590976/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828920.514608:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828920.514610:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828920.514612:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790590976:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828920.514616:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790591104:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787828920.514625:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828920.514631:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828920/real 1787828920] req@ffff9e680a2d7c00 x1874668790591104/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828920.514639:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790591104/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828920.514648:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828920.514651:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828920.514664:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790591104:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828920.514670:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790591232:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828920.514690:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828920.514700:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828920/real 1787828920] req@ffff9e680a2d70c0 x1874668790591232/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828920.514707:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790591232/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828920.514713:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828920.514715:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828920.514717:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790591232:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828920.514737:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790591360:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787828920.514788:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828920.514816:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828920/real 1787828920] req@ffff9e680a2d6580 x1874668790591360/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828920.514826:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790591360/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828920.514834:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828920.514837:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828920.514840:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790591360:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828921.025865:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828921.217907:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828922.049853:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828922.241843:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828923.073775:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828923.265981:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828924.097837:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828924.289956:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828925.121850:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828925.313896:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828925.633864:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828925.633874:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828925.633880:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828925.633892:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5562 00000100:00080000:1.0:1787828925.633905:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828925.633914:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828925.633958:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790591488/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828925.633994:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828925.633998:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828925.634001:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828925.634005:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5562 00000100:00080000:1.0:1787828925.634010:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828925.634015:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828925.634027:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828925.634040:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790591616/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828925.634053:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828925.634056:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828925.634062:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828925.634064:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790591488:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828925.634070:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5562 00000100:00080000:1.0:1787828925.634081:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828925.634086:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828925.634091:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828925.634108:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790591744/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828925.634118:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828925.634122:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828925.634125:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828925.634130:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5562 00000100:00080000:1.0:1787828925.634144:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828925.634149:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787828925.634156:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1787828925.634158:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828925.634197:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790591872/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828925.634217:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828925.634221:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828925.634224:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828925.634231:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5562 00000100:00100000:0.0:1787828925.634233:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828925/real 1787828925] req@ffff9e6831aa9e00 x1874668790591488/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828925.634237:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828925.634248:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828925.634259:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828925.634274:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790591488/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828925.634292:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790592000/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828925.634293:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828925.634307:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828925.634322:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790591488:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828925.634322:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828925.634338:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828925.634341:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828925.634347:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5562 00000100:00080000:1.0:1787828925.634352:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828925.634356:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828925.634361:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828925.634376:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790592128/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828925.634387:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828925.634388:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790591616:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828925.634391:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828925.634396:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828925.634408:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5562 00000100:00080000:1.0:1787828925.634412:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828925.634416:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828925.634421:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828925.634435:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790592256/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828925.634446:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828925.634449:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828925.634451:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787828925.634460:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828925.634461:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5562 00000100:00080000:1.0:1787828925.634472:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828925.634476:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828925.634482:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828925.634495:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790592384/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828925.634505:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828925/real 1787828925] req@ffff9e6831aa9680 x1874668790591616/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828925.634508:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828925.634515:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828925.634517:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828925.634522:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5562 00000100:00080000:1.0:1787828925.634527:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828925.634538:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828925.634548:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828925.634560:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790591616/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828925.634562:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790592512/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828925.634607:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828925.634621:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828925.634629:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790591616:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828925.634639:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790591744:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828925.634675:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828925.634689:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828925/real 1787828925] req@ffff9e6831aaa580 x1874668790591744/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828925.634756:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790591744/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828925.634767:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828925.634772:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828925.634776:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790591744:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828925.634785:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790591872:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828925.634813:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828925.634827:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828925/real 1787828925] req@ffff9e6831aa8f00 x1874668790591872/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828925.634841:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790591872/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828925.634859:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828925.634863:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828925.634867:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790591872:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828925.634874:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790592000:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828925.634893:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828925.634904:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828925/real 1787828925] req@ffff9e6831aaa940 x1874668790592000/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828925.634918:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790592000/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828925.634928:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828925.634931:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828925.634935:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790592000:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828925.634957:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790592128:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828925.634996:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828925.635015:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828925/real 1787828925] req@ffff9e6831aa9a40 x1874668790592128/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828925.635036:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790592128/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828925.635053:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828925.635058:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828925.635066:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790592128:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828925.635077:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790592256:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828925.635124:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828925.635140:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828925/real 1787828925] req@ffff9e6831aabc00 x1874668790592256/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828925.635155:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790592256/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828925.635166:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828925.635170:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828925.635215:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790592256:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828925.635224:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790592384:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828925.635247:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828925.635260:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828925/real 1787828925] req@ffff9e6831aab480 x1874668790592384/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828925.635273:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790592384/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828925.635281:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828925.635284:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828925.635288:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790592384:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828925.635293:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790592512:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828925.635309:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828925.635319:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828925/real 1787828925] req@ffff9e6831aaad00 x1874668790592512/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828925.635331:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790592512/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828925.635339:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828925.635342:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828925.635345:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790592512:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828926.145796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828926.337851:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828927.169802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828927.361837:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828928.193764:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828928.385850:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828929.217802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828929.409863:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828930.241786:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828930.433882:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828930.753825:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828930.753836:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828930.753840:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828930.753847:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5567 00000100:00080000:1.0:1787828930.753863:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828930.753869:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828930.753901:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790592640/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828930.753924:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828930.753929:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828930.753930:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828930.753933:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5567 00000100:00080000:1.0:1787828930.753935:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828930.753938:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828930.753945:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828930.753955:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790592768/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828930.753963:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828930.753966:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828930.753967:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828930.753970:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5567 00000100:00080000:1.0:1787828930.753975:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828930.753976:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790592640:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828930.753979:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828930.753996:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828930.754005:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790592896/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828930.754012:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828930.754014:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828930.754016:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828930.754018:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5567 00000100:00080000:1.0:1787828930.754020:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828930.754023:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828930.754026:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828930.754034:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790593024/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828930.754040:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828930.754043:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828930.754044:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828930.754046:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5567 00000100:00080000:1.0:1787828930.754048:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828930.754052:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787828930.754057:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1787828930.754057:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828930.754069:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790593152/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828930.754081:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828930.754084:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828930.754086:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828930.754090:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5567 00000100:00080000:1.0:1787828930.754094:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828930.754099:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828930.754104:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828930/real 1787828930] req@ffff9e6831aa8b40 x1874668790592640/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828930.754107:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828930.754125:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790593280/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828930.754131:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790592640/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828930.754156:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828930.754161:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828930.754163:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828930.754171:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5567 00000100:00080000:0.0:1787828930.754172:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828930.754175:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828930.754180:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828930.754185:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828930.754189:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790592640:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787828930.754193:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828930.754246:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790592768:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828930.754251:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790593408/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828930.754266:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828930.754269:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828930.754272:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828930.754276:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5567 00000100:00080000:1.0:1787828930.754282:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828930.754290:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828930.754291:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828930.754302:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828930.754314:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790593536/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828930.754336:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828930/real 1787828930] req@ffff9e6831aa8780 x1874668790592768/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828930.754337:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828930.754346:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828930.754351:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828930.754360:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5567 00000100:00080000:1.0:1787828930.754366:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828930.754372:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828930.754378:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828930.754400:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790592768/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828930.754407:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790593664/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828930.754436:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828930.754440:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828930.754444:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790592768:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828930.754449:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790592896:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828930.754466:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828930.754474:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828930/real 1787828930] req@ffff9e6831aaa1c0 x1874668790592896/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828930.754482:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790592896/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828930.754502:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828930.754517:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828930.754520:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790592896:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828930.754527:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790593024:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828930.754548:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828930.754556:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828930/real 1787828930] req@ffff9e6831aa8000 x1874668790593024/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828930.754564:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790593024/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828930.754580:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828930.754582:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828930.754584:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790593024:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828930.754588:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790593152:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828930.754599:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828930.754605:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828930/real 1787828930] req@ffff9e6831aaad00 x1874668790593152/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828930.754612:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790593152/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828930.754617:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828930.754618:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828930.754620:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790593152:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828930.754624:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790593280:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828930.754633:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828930.754639:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828930/real 1787828930] req@ffff9e6831aab480 x1874668790593280/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828930.754646:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790593280/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828930.754652:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828930.754653:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828930.754655:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790593280:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828930.754667:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790593408:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828930.754695:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828930.754706:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828930/real 1787828930] req@ffff9e6831aabc00 x1874668790593408/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828930.754751:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790593408/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828930.754759:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828930.754761:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828930.754765:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790593408:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828930.754770:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790593536:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828930.754789:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828930.754799:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828930/real 1787828930] req@ffff9e6831aa9a40 x1874668790593536/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828930.754809:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790593536/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828930.754817:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828930.754819:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828930.754822:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790593536:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828930.754827:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790593664:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828930.754840:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828930.754848:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828930/real 1787828930] req@ffff9e6831aaa940 x1874668790593664/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828930.754858:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790593664/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828930.754866:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828930.754869:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828930.754872:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790593664:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828931.265815:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828931.457926:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828932.289789:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828932.481937:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828933.313765:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828933.505874:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828934.337779:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828934.529844:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828935.361796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828935.553817:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828935.873765:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828935.873773:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828935.873776:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828935.873784:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5572 00000100:00080000:3.0:1787828935.873792:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828935.873798:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787828935.873827:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790593792/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828935.873854:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828935.873859:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828935.873861:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828935.873866:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5572 00000100:00080000:3.0:1787828935.873871:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828935.873876:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828935.873884:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828935.873894:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790593792:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787828935.873899:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790593920/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828935.873909:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828935.873911:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828935.873913:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828935.873916:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5572 00000100:00080000:3.0:1787828935.873919:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828935.873922:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828935.873925:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828935.873934:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790594048/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828935.873942:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828935.873944:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828935.873946:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828935.873948:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5572 00000800:00000100:0.0:1787828935.873952:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828935.873952:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828935.873959:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828935.873963:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828935.873971:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790594176/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828935.873978:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828935.873980:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828935.873982:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828935.873985:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5572 00000100:00100000:0.0:1787828935.873990:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828935/real 1787828935] req@ffff9e680a2d52c0 x1874668790593792/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828935.873991:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828935.873998:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828935.874006:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828935.874015:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790593792/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828935.874018:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790594304/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828935.874038:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828935.874039:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828935.874045:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828935.874046:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828935.874048:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828935.874053:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790593792:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787828935.874057:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5572 00000100:00080000:3.0:1787828935.874061:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828935.874064:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828935.874067:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828935.874074:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790594432/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828935.874081:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828935.874083:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787828935.874084:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828935.874087:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5572 00000100:00080000:3.0:1787828935.874090:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828935.874095:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790593920:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828935.874095:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828935.874114:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828935.874126:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790594560/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828935.874142:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828935.874144:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1787828935.874151:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787828935.874153:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828935.874159:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5572 00000100:00080000:3.0:1787828935.874165:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828935.874167:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828935/real 1787828935] req@ffff9e680a2d4000 x1874668790593920/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828935.874172:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828935.874180:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828935.874187:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790593920/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828935.874192:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790594688/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828935.874229:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828935.874230:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828935.874237:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787828935.874237:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828935.874241:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828935.874243:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790593920:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828935.874248:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5572 00000100:00080000:3.0:1787828935.874251:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828935.874252:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790594048:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787828935.874257:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787828935.874262:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787828935.874270:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790594816/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828935.874282:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828935.874292:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828935/real 1787828935] req@ffff9e680a2d4780 x1874668790594048/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828935.874300:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790594048/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828935.874306:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828935.874309:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828935.874311:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790594048:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828935.874315:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790594176:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828935.874325:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828935.874331:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828935/real 1787828935] req@ffff9e680a2d5a40 x1874668790594176/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828935.874338:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790594176/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828935.874343:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828935.874345:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828935.874347:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790594176:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828935.874351:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790594304:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828935.874361:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828935.874366:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828935/real 1787828935] req@ffff9e680a2d43c0 x1874668790594304/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828935.874373:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790594304/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828935.874378:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828935.874380:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828935.874382:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790594304:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828935.874385:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790594432:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828935.874394:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828935.874400:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828935/real 1787828935] req@ffff9e680a2d6580 x1874668790594432/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828935.874407:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790594432/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828935.874413:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828935.874415:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828935.874417:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790594432:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828935.874430:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790594560:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828935.874440:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828935.874446:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828935/real 1787828935] req@ffff9e680a2d70c0 x1874668790594560/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828935.874458:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790594560/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828935.874463:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828935.874465:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828935.874467:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790594560:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828935.874477:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790594688:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828935.874493:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828935.874504:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828935/real 1787828935] req@ffff9e680a2d7c00 x1874668790594688/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828935.874516:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790594688/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828935.874525:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828935.874528:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828935.874532:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790594688:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828935.874538:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790594816:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828935.874557:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828935.874568:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828935/real 1787828935] req@ffff9e680a2d4f00 x1874668790594816/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828935.874575:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790594816/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828960 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828935.874585:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828935.874587:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828935.874589:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790594816:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828936.385766:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828936.577893:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828937.409814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828937.601869:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828938.433778:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828938.625851:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828939.457861:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828939.649857:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828940.481770:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828940.673941:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828940.993823:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828940.993835:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828940.993840:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828940.993853:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5577 00000100:00080000:1.0:1787828940.993866:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828940.993874:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828940.993934:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790594944/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828940.993970:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828940.993973:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828940.993975:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828940.993978:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5577 00000100:00080000:1.0:1787828940.993981:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828940.993984:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828940.993994:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828940.994003:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790595072/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828940.994011:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828940.994014:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828940.994015:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828940.994018:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5577 00000100:00080000:1.0:1787828940.994020:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828940.994022:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828940.994025:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828940.994033:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790595200/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828940.994040:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828940.994043:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828940.994044:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828940.994047:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5577 00000100:00080000:1.0:1787828940.994051:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828940.994054:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828940.994059:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828940.994064:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790594944:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828940.994082:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790595328/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828940.994109:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828940.994111:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828940.994113:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828940.994116:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5577 00000100:00080000:1.0:1787828940.994119:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828940.994121:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828940.994125:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828940.994137:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790595456/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828940.994144:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828940.994147:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828940.994148:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828940.994152:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5577 00000100:00080000:1.0:1787828940.994180:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828940.994184:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828940.994187:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828940.994194:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828940.994198:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790595584/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828940.994212:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828940.994215:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828940.994217:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828940.994220:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5577 00000100:00080000:1.0:1787828940.994223:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828940.994226:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828940.994230:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828940.994237:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790595712/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828940.994243:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828940.994245:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828940.994247:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828940.994251:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5577 00000100:00080000:1.0:1787828940.994254:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828940.994259:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828940/real 1787828940] req@ffff9e6831aa8f00 x1874668790594944/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828940.994260:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828940.994266:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828940.994279:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790595840/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828940.994300:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790594944/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828940.994300:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828940.994307:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828940.994311:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828940.994321:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5577 00000100:00080000:1.0:1787828940.994325:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828940.994326:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828940.994329:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828940.994336:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787828940.994340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787828940.994346:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790595968/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828940.994351:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790594944:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787828940.994368:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790595072:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828940.994411:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828940.994424:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828940/real 1787828940] req@ffff9e6831aaa580 x1874668790595072/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828940.994435:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790595072/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828940.994443:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828940.994446:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828940.994449:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790595072:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828940.994453:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790595200:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828940.994479:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828940.994487:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828940/real 1787828940] req@ffff9e6831aa9680 x1874668790595200/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828940.994495:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790595200/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828940.994502:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828940.994504:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828940.994507:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790595200:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828940.994539:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790595328:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828940.994553:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828940.994561:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828940/real 1787828940] req@ffff9e6831aa9e00 x1874668790595328/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828940.994568:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790595328/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828940.994574:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828940.994576:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828940.994580:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790595328:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828940.994584:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790595456:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828940.994595:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828940.994601:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828940/real 1787828940] req@ffff9e6831aaa940 x1874668790595456/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828940.994609:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790595456/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828940.994632:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828940.994634:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828940.994636:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790595456:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828940.994641:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790595584:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828940.994654:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828940.994661:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828940/real 1787828940] req@ffff9e6831aa9a40 x1874668790595584/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828940.994670:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790595584/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828940.994677:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828940.994679:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828940.994681:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790595584:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828940.994685:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790595712:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828940.994743:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828940.994753:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828940/real 1787828940] req@ffff9e6831aabc00 x1874668790595712/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828940.994762:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790595712/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828940.994768:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828940.994771:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828940.994774:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790595712:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828940.994778:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790595840:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828940.994791:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828940.994809:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828940/real 1787828940] req@ffff9e6831aab480 x1874668790595840/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828940.994818:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790595840/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828940.994824:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828940.994826:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828940.994828:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790595840:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828940.994832:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790595968:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828940.994843:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828940.994850:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828940/real 1787828940] req@ffff9e6831aaad00 x1874668790595968/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828940.994858:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790595968/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828965 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828940.994864:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828940.994867:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828940.994869:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790595968:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828941.505812:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828941.697831:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828942.529851:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828942.721869:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828943.553856:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828943.745858:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828944.577848:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828944.769844:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828945.601880:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828945.794005:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828946.113790:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828946.113801:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828946.113805:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828946.113841:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5582 00000100:00080000:1.0:1787828946.113854:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828946.113862:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828946.113910:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790596096/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828946.113943:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828946.113948:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828946.113950:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828946.113956:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5582 00000100:00080000:1.0:1787828946.113961:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828946.113966:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828946.113979:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828946.113993:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790596224/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828946.114011:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828946.114013:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790596096:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828946.114017:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828946.114024:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828946.114071:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5582 00000100:00080000:1.0:1787828946.114074:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828946.114078:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828946.114082:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828946.114092:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790596352/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828946.114101:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828946.114104:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828946.114105:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828946.114109:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5582 00000100:00080000:1.0:1787828946.114135:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828946.114138:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828946.114143:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828946.114149:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787828946.114154:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790596480/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828946.114162:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828946.114166:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828946.114168:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828946.114173:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5582 00000100:00080000:1.0:1787828946.114182:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828946.114185:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828946.114189:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828946.114200:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790596608/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828946.114208:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828946.114213:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828946/real 1787828946] req@ffff9e6831aa8000 x1874668790596096/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828946.114213:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828946.114215:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828946.114233:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5582 00000100:00080000:1.0:1787828946.114245:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828946.114253:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828946.114259:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828946.114265:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790596096/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828946.114271:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790596736/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828946.114316:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828946.114318:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828946.114333:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787828946.114339:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828946.114340:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828946.114353:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790596096:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828946.114355:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5582 00000100:00080000:1.0:1787828946.114367:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828946.114374:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828946.114381:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828946.114391:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790596224:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828946.114400:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790596864/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828946.114416:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828946.114419:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828946.114421:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787828946.114429:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828946.114430:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5582 00000100:00080000:1.0:1787828946.114442:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828946.114446:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828946.114451:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828946.114454:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828946/real 1787828946] req@ffff9e6831aaa1c0 x1874668790596224/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828946.114466:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790596992/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828946.114474:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790596224/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828946.114488:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828946.114493:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828946.114496:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828946.114502:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828946.114504:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828946.114510:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790596224:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828946.114518:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5582 00000100:00100000:0.0:1787828946.114524:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790596352:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828946.114526:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828946.114542:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828946.114550:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828946.114557:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828946.114572:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828946/real 1787828946] req@ffff9e6831aa8780 x1874668790596352/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828946.114578:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790597120/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828946.114604:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790596352/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828946.114618:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828946.114622:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828946.114625:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790596352:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828946.114632:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790596480:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828946.114651:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828946.114659:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828946/real 1787828946] req@ffff9e6831aa8b40 x1874668790596480/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828946.114673:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790596480/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828946.114684:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828946.114691:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828946.114734:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790596480:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828946.114747:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790596608:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828946.114777:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828946.114793:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828946/real 1787828946] req@ffff9e6831aaad00 x1874668790596608/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828946.114814:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790596608/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828946.114823:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828946.114828:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828946.114836:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790596608:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828946.114846:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790596736:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828946.114867:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828946.114887:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828946/real 1787828946] req@ffff9e6831aab480 x1874668790596736/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828946.114905:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790596736/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828946.114919:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828946.114926:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828946.114931:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790596736:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828946.114953:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790596864:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828946.114972:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828946.114988:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828946/real 1787828946] req@ffff9e6831aabc00 x1874668790596864/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828946.115001:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790596864/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828946.115011:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828946.115016:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828946.115021:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790596864:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828946.115030:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790596992:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828946.115073:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828946.115084:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828946/real 1787828946] req@ffff9e6831aa9a40 x1874668790596992/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828946.115094:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790596992/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828946.115101:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828946.115104:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828946.115107:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790596992:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828946.115112:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790597120:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828946.115129:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828946.115137:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828946/real 1787828946] req@ffff9e6831aaa940 x1874668790597120/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828946.115147:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790597120/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828946.115156:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828946.115160:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828946.115162:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790597120:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828946.625821:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828946.817867:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828947.649771:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828947.841836:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828948.673786:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828948.865865:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828949.697777:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828949.889836:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828950.721817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828950.913890:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828951.233756:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828951.233765:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828951.233769:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828951.233777:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5587 00000100:00080000:1.0:1787828951.233786:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828951.233795:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828951.233846:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790597248/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828951.233874:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828951.233877:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828951.233878:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828951.233881:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5587 00000100:00080000:1.0:1787828951.233884:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828951.233887:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828951.233895:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828951.233906:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790597376/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828951.233914:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828951.233918:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828951.233919:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828951.233923:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5587 00000100:00100000:0.0:1787828951.233925:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790597248:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828951.233926:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828951.233932:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828951.233936:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828951.233945:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790597504/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828951.233951:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828951.233953:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828951.233955:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828951.233958:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5587 00000100:00080000:1.0:1787828951.233961:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828951.233964:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828951.233978:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828951.233985:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790597632/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828951.233991:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828951.233994:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828951.234006:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828951.234012:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5587 00000800:00000100:0.0:1787828951.234014:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828951.234017:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828951.234022:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828951.234026:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828951.234036:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790597760/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828951.234043:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828951.234046:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828951.234047:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828951.234050:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5587 00000100:00080000:1.0:1787828951.234053:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828951.234055:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828951/real 1787828951] req@ffff9e6831aa9e00 x1874668790597248/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828951.234058:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828951.234064:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828951.234078:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790597888/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828951.234082:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790597248/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828951.234092:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828951.234098:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828951.234099:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828951.234100:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828951.234104:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5587 00000100:00080000:0.0:1787828951.234109:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828951.234110:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828951.234116:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828951.234118:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790597248:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787828951.234121:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828951.234130:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790598016/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828951.234136:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828951.234138:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828951.234140:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828951.234144:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790597376:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828951.234144:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5587 00000100:00080000:1.0:1787828951.234146:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828951.234153:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828951.234157:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828951.234166:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790598144/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828951.234173:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828951.234176:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828951.234177:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787828951.234191:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828951.234193:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5587 00000100:00080000:1.0:1787828951.234198:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828951.234200:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828951.234204:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828951.234212:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828951/real 1787828951] req@ffff9e6831aa9680 x1874668790597376/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828951.234214:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790598272/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828951.234241:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790597376/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828951.234270:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828951.234273:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828951.234275:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790597376:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828951.234280:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790597504:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828951.234295:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828951.234302:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828951/real 1787828951] req@ffff9e6831aaa580 x1874668790597504/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828951.234320:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790597504/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828951.234325:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828951.234327:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828951.234329:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790597504:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828951.234333:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790597632:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828951.234343:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828951.234349:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828951/real 1787828951] req@ffff9e6831aa8f00 x1874668790597632/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828951.234355:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790597632/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828951.234361:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828951.234362:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828951.234365:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790597632:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828951.234369:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790597760:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828951.234389:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828951.234395:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828951/real 1787828951] req@ffff9e6831aaa940 x1874668790597760/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828951.234401:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790597760/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828951.234407:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828951.234408:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828951.234410:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790597760:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828951.234413:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790597888:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828951.234422:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828951.234428:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828951/real 1787828951] req@ffff9e6831aa9a40 x1874668790597888/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828951.234435:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790597888/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828951.234454:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828951.234457:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828951.234461:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790597888:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828951.234467:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790598016:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828951.234486:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828951.234498:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828951/real 1787828951] req@ffff9e6831aabc00 x1874668790598016/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828951.234520:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790598016/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828951.234526:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828951.234528:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828951.234530:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790598016:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828951.234544:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790598144:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828951.234558:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828951.234565:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828951/real 1787828951] req@ffff9e6831aab480 x1874668790598144/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828951.234583:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790598144/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828951.234588:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828951.234590:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828951.234592:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790598144:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828951.234595:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790598272:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828951.234612:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828951.234619:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828951/real 1787828951] req@ffff9e6831aaad00 x1874668790598272/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828951.234626:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790598272/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828951.234631:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828951.234633:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828951.234646:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790598272:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787828951.745781:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828951.937847:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828952.769855:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828952.961881:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828953.793842:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828953.985858:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828954.817822:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828955.009859:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828955.841828:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828956.033891:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828956.353808:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828956.353817:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828956.353820:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828956.353828:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5593 00000100:00080000:2.0:1787828956.353837:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828956.353842:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828956.353873:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790598400/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828956.353895:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828956.353898:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828956.353899:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828956.353902:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5593 00000100:00080000:2.0:1787828956.353904:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828956.353908:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828956.353917:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828956.353930:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790598400:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787828956.353930:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790598528/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828956.353945:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828956.353949:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828956.353951:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828956.353956:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5593 00000100:00080000:2.0:1787828956.353975:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828956.353981:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828956.353987:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828956.354000:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790598656/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828956.354022:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828956.354024:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828956.354028:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828956.354030:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828956.354034:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5593 00000100:00080000:2.0:1787828956.354039:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828956.354043:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828956.354047:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828956.354058:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790598784/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828956.354065:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828956.354070:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828956/real 1787828956] req@ffff9e6804420b40 x1874668790598400/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828956.354070:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828956.354084:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828956.354092:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5593 00000100:00080000:2.0:1787828956.354096:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828956.354100:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828956.354104:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790598400/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828956.354105:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787828956.354117:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787828956.354120:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790598912/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828956.354123:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828956.354130:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790598400:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828956.354137:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828956.354141:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828956.354142:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828956.354146:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5593 00000100:00080000:2.0:1787828956.354148:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828956.354151:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828956.354154:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828956.354156:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790598528:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828956.354166:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790599040/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828956.354186:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787828956.354190:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828956.354194:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828956.354198:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828956.354203:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828956/real 1787828956] req@ffff9e68044230c0 x1874668790598528/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828956.354205:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5593 00000100:00080000:2.0:1787828956.354213:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828956.354218:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790598528/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828956.354223:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787828956.354234:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828956.354237:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828956.354240:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790598528:192.168.203.109@tcp:38:kworker.0 00000008:00100000:2.0:1787828956.354242:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828956.354246:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790598656:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828956.354262:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790599168/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828956.354266:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828956.354276:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828956/real 1787828956] req@ffff9e68044212c0 x1874668790598656/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828956.354280:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828956.354287:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787828956.354288:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790598656/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828956.354293:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828956.354297:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828956.354306:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828956.354311:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790598656:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828956.354314:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5593 00000100:00100000:0.0:1787828956.354318:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790598784:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787828956.354322:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828956.354330:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787828956.354334:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:2.0:1787828956.354336:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828956.354343:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828956/real 1787828956] req@ffff9e6804420780 x1874668790598784/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828956.354351:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790599296/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828956.354356:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790598784/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828956.354376:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828956.354379:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828956.354380:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828956.354384:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828956.354385:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787828956.354390:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5593 00000100:00100000:0.0:1787828956.354392:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790598784:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787828956.354395:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828956.354399:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790598912:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787828956.354404:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828956.354411:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828956.354416:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828956.354425:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828956/real 1787828956] req@ffff9e6804423840 x1874668790598912/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828956.354430:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790599424/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828956.354438:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790598912/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828956.354454:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828956.354458:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828956.354461:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790598912:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828956.354475:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790599040:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828956.354485:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828956.354492:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828956/real 1787828956] req@ffff9e68044221c0 x1874668790599040/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828956.354499:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790599040/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828956.354505:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828956.354507:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828956.354508:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790599040:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828956.354524:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790599168:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828956.354540:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828956.354551:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828956/real 1787828956] req@ffff9e6804422d00 x1874668790599168/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828956.354563:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790599168/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828956.354588:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828956.354592:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828956.354594:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790599168:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828956.354599:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790599296:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828956.354613:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828956.354621:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828956/real 1787828956] req@ffff9e6804420f00 x1874668790599296/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828956.354628:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790599296/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828956.354633:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828956.354643:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828956.354646:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790599296:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828956.354649:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790599424:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828956.354661:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828956.354667:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828956/real 1787828956] req@ffff9e6804421e00 x1874668790599424/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828956.354674:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790599424/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828956.354679:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828956.354680:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828956.354683:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790599424:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828956.865866:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828957.057859:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828957.889793:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828958.081916:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828958.913774:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828959.105842:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828959.937809:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828960.129837:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828960.961829:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828961.153899:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828961.473770:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828961.473778:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828961.473782:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828961.473789:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5598 00000100:00080000:2.0:1787828961.473799:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828961.473805:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828961.473836:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790599552/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828961.473858:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828961.473861:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828961.473862:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828961.473875:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5598 00000100:00080000:2.0:1787828961.473878:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828961.473881:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828961.473890:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828961.473900:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790599552:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787828961.473901:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790599680/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828961.473915:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828961.473917:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828961.473918:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828961.473921:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5598 00000100:00080000:2.0:1787828961.473923:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828961.473938:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828961.473942:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828961.473954:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790599808/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828961.473961:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828961.473963:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828961.473964:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828961.473967:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5598 00000100:00080000:2.0:1787828961.473981:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828961.473986:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828961.473986:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828961.473993:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828961.474002:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790599936/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828961.474008:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828961.474011:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828961.474012:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828961.474015:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5598 00000100:00080000:2.0:1787828961.474017:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828961.474020:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828961.474024:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828961.474028:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828961/real 1787828961] req@ffff9e6804422940 x1874668790599552/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828961.474035:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790600064/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828961.474054:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828961.474057:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828961.474058:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828961.474061:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5598 00000100:00080000:2.0:1787828961.474064:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828961.474066:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790599552/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828961.474070:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828961.474076:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828961.474087:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790600192/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828961.474088:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828961.474097:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787828961.474099:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828961.474103:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790599552:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828961.474104:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828961.474108:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828961.474122:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5598 00000100:00080000:2.0:1787828961.474124:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828961.474127:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828961.474130:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828961.474136:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790600320/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828961.474142:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790599680:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828961.474145:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828961.474148:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828961.474149:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828961.474153:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5598 00000100:00080000:2.0:1787828961.474156:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828961.474159:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828961.474163:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828961.474168:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828961.474172:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790600448/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828961.474181:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828961.474182:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828961/real 1787828961] req@ffff9e6804423480 x1874668790599680/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828961.474188:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828961.474192:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828961.474200:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5598 00000100:00080000:2.0:1787828961.474203:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828961.474204:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790599680/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828961.474208:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828961.474215:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787828961.474219:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828961.474225:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828961.474225:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790600576/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828961.474232:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790599680:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828961.474245:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790599808:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828961.474260:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828961.474267:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828961/real 1787828961] req@ffff9e6804421680 x1874668790599808/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828961.474275:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790599808/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828961.474280:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828961.474282:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828961.474284:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790599808:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828961.474287:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790599936:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828961.474297:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828961.474303:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828961/real 1787828961] req@ffff9e6804422580 x1874668790599936/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828961.474310:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790599936/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828961.474316:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828961.474317:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828961.474319:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790599936:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828961.474322:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790600064:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828961.474331:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828961.474337:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828961/real 1787828961] req@ffff9e6804421e00 x1874668790600064/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828961.474344:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790600064/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828961.474349:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828961.474351:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828961.474353:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790600064:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828961.474357:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790600192:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828961.474366:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828961.474371:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828961/real 1787828961] req@ffff9e6804420f00 x1874668790600192/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828961.474377:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790600192/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828961.474383:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828961.474385:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828961.474386:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790600192:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828961.474398:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790600320:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828961.474408:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828961.474414:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828961/real 1787828961] req@ffff9e6804422d00 x1874668790600320/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828961.474421:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790600320/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828961.474426:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828961.474427:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828961.474430:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790600320:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828961.474434:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790600448:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828961.474443:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828961.474448:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828961/real 1787828961] req@ffff9e68044221c0 x1874668790600448/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828961.474455:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790600448/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828961.474460:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828961.474462:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828961.474464:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790600448:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828961.474467:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790600576:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828961.474476:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828961.474481:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828961/real 1787828961] req@ffff9e6804423840 x1874668790600576/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828961.474487:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790600576/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828961.474492:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828961.474494:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828961.474496:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790600576:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828961.985941:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828962.177959:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828963.009832:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828963.201844:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828964.033790:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828964.225867:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828965.057825:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828965.249892:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828966.081829:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828966.273881:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828966.593765:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828966.593784:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828966.593788:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828966.593797:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5603 00000100:00080000:2.0:1787828966.593805:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828966.593811:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828966.593852:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790600704/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828966.593874:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828966.593877:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828966.593878:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828966.593881:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5603 00000100:00080000:2.0:1787828966.593883:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828966.593886:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828966.593905:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828966.593915:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790600832/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828966.593924:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828966.593929:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790600704:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828966.593929:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828966.593932:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828966.593936:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5603 00000100:00080000:2.0:1787828966.593938:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828966.593941:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828966.593948:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828966.593956:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790600960/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828966.593963:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828966.593965:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828966.593966:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828966.593968:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5603 00000100:00080000:2.0:1787828966.593970:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828966.593973:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828966.593976:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828966.593983:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790601088/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828966.593989:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828966.593991:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828966.593993:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828966.593996:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5603 00000100:00080000:2.0:1787828966.593999:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828966.594001:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828966.594004:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828966.594012:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790601216/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828966.594017:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828966.594020:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828966.594021:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828966.594024:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5603 00000100:00080000:2.0:1787828966.594025:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828966.594028:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828966.594031:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828966.594040:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790601344/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828966.594045:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828966.594047:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828966.594048:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828966.594052:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5603 00000100:00100000:0.0:1787828966.594054:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790600832:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828966.594055:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828966.594061:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828966.594066:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828966.594076:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790601472/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828966.594093:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828966.594096:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828966.594097:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828966.594100:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5603 00000100:00080000:2.0:1787828966.594102:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828966.594104:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828966.594108:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828966.594113:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790600960:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828966.594121:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790601600/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828966.594129:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828966.594131:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828966.594133:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828966.594136:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5603 00000100:00080000:2.0:1787828966.594139:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828966.594153:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828966.594156:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828966.594162:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790601728/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828966.594171:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790601088:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828966.594202:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790601216:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828966.594235:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790601344:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828966.594271:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790601472:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828966.594295:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790601600:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828966.594326:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790601728:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828967.105829:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828967.297831:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828968.129774:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828968.321856:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828969.153801:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828969.345881:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787828969.666013:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:3.0:1787828969.666064:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787828969.666084:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787828969.666148:0:2386:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:3.0:1787828969.666165:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787828969.666177:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787828969.666188:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1787828969.666189:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828966/real 1787828969] req@ffff9e6804421e00 x1874668790601728/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787828969.666202:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787828969.666217:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1787828969.666238:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790601728/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787828969.666239:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787828969.666253:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00080000:0.0:1787828969.666258:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828969.666267:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828969.666273:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790601728:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828969.666283:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828966/real 1787828969] req@ffff9e68044212c0 x1874668790600832/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828969.666297:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790600832/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828969.666307:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828969.666312:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828969.666316:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790600832:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828969.666323:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828966/real 1787828969] req@ffff9e68044230c0 x1874668790600960/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828969.666335:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790600960/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828969.666342:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828969.666347:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828969.666350:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790600960:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828969.666357:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828966/real 1787828969] req@ffff9e6804420b40 x1874668790601088/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828969.666368:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790601088/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828969.666376:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828969.666380:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828969.666383:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790601088:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828969.666406:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828966/real 1787828969] req@ffff9e6804423840 x1874668790601216/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828969.666420:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790601216/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828969.666429:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828969.666433:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828969.666437:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790601216:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828969.666445:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828966/real 1787828969] req@ffff9e68044221c0 x1874668790601344/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828969.666458:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790601344/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828969.666468:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828969.666471:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828969.666473:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790601344:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828969.666479:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828966/real 1787828969] req@ffff9e6804420780 x1874668790600704/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828969.666493:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790600704/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828969.666501:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828969.666504:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828969.666508:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790600704:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787828969.666544:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828966/real 1787828969] req@ffff9e6804422d00 x1874668790601472/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828969.666573:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790601472/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828969.666582:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828969.666587:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828969.666590:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790601472:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828969.666597:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828966/real 1787828969] req@ffff9e6804420f00 x1874668790601600/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828969.666609:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790601600/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828969.666617:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828969.666620:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828969.666624:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790601600:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828970.177828:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828970.369909:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828971.201767:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828971.393847:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828971.713779:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828971.713787:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828971.713791:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828971.713801:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5608 00000100:00080000:2.0:1787828971.713809:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828971.713815:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828971.713844:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790601856/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828971.713877:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828971.713880:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828971.713881:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828971.713884:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5608 00000100:00080000:2.0:1787828971.713886:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828971.713889:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828971.713897:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828971.713907:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790601984/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828971.713915:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828971.713920:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787828971.713921:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790601856:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828971.713923:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828971.713932:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5608 00000100:00080000:2.0:1787828971.713934:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828971.713936:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828971.713939:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828971.713948:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790602112/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828971.713960:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828971.713962:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828971.713963:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828971.713966:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5608 00000100:00080000:2.0:1787828971.713968:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828971.713971:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828971.713974:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828971.713982:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828971.713983:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790602240/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828971.713992:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828971.713995:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828971.713997:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828971.714002:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5608 00000100:00080000:2.0:1787828971.714005:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828971.714009:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828971.714014:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828971.714028:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828971/real 1787828971] req@ffff9e6804422580 x1874668790601856/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828971.714031:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790602368/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828971.714066:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828971.714071:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790601856/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828971.714072:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828971.714077:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828971.714087:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5608 00000100:00080000:0.0:1787828971.714093:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828971.714094:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828971.714104:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787828971.714105:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828971.714114:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790601856:192.168.203.109@tcp:250:kworker.0 00000008:00100000:2.0:1787828971.714114:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828971.714141:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790602496/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828971.714146:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790601984:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828971.714160:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828971.714164:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828971.714166:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828971.714171:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5608 00000100:00080000:2.0:1787828971.714176:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828971.714182:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787828971.714187:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:2.0:1787828971.714190:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828971.714205:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790602624/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828971.714207:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828971/real 1787828971] req@ffff9e6804421680 x1874668790601984/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828971.714240:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828971.714245:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787828971.714250:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790601984/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828971.714250:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828971.714259:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5608 00000100:00080000:2.0:1787828971.714267:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828971.714273:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828971.714275:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787828971.714283:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:2.0:1787828971.714284:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828971.714297:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790601984:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828971.714300:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790602752/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828971.714310:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790602112:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828971.714328:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828971.714332:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828971.714334:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828971.714339:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5608 00000100:00080000:2.0:1787828971.714344:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828971.714347:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828971.714353:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828971.714360:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828971.714368:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828971/real 1787828971] req@ffff9e6804423480 x1874668790602112/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828971.714376:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790602880/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828971.714396:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790602112/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828971.714408:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828971.714412:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828971.714417:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790602112:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828971.714423:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790602240:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828971.714446:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828971.714458:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828971/real 1787828971] req@ffff9e6804422940 x1874668790602240/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828971.714479:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790602240/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828971.714494:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828971.714498:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828971.714501:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790602240:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828971.714508:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790602368:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828971.714529:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828971.714543:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828971/real 1787828971] req@ffff9e6804420f00 x1874668790602368/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828971.714554:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790602368/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828971.714562:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828971.714565:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828971.714568:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790602368:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828971.714587:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790602496:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828971.714608:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828971.714621:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828971/real 1787828971] req@ffff9e6804422d00 x1874668790602496/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828971.714633:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790602496/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828971.714643:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828971.714646:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828971.714650:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790602496:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828971.714657:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790602624:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828971.714675:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828971.714686:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828971/real 1787828971] req@ffff9e6804420780 x1874668790602624/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828971.714722:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790602624/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828971.714731:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828971.714734:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828971.714737:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790602624:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828971.714744:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790602752:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828971.714764:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828971.714777:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828971/real 1787828971] req@ffff9e68044221c0 x1874668790602752/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828971.714790:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790602752/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828971.714799:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828971.714802:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828971.714806:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790602752:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828971.714812:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790602880:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828971.714832:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828971.714845:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828971/real 1787828971] req@ffff9e6804423840 x1874668790602880/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828971.714871:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790602880/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828971.714880:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828971.714883:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828971.714887:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790602880:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828972.225780:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828972.417974:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828973.249788:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828973.441984:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828974.273867:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828974.465876:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828975.297761:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828975.489888:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828976.321809:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828976.513867:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828976.833804:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828976.833814:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828976.833828:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828976.833838:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5613 00000100:00080000:1.0:1787828976.833849:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828976.833856:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828976.833893:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790603008/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828976.833920:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828976.833924:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828976.833926:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828976.833930:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5613 00000100:00080000:1.0:1787828976.833934:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828976.833938:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828976.833949:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828976.833964:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790603136/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828976.833976:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828976.833981:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790603008:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828976.833981:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828976.833984:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828976.833994:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5613 00000100:00080000:1.0:1787828976.833997:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828976.834001:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828976.834006:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828976.834017:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790603264/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828976.834028:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828976.834032:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828976.834034:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828976.834038:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5613 00000100:00080000:1.0:1787828976.834042:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828976.834047:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828976.834047:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828976.834055:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828976.834068:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790603392/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828976.834078:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828976.834082:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828976.834086:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828976.834087:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828976/real 1787828976] req@ffff9e6831aa8b40 x1874668790603008/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828976.834095:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5613 00000100:00080000:1.0:1787828976.834103:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828976.834108:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790603008/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828976.834111:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787828976.834123:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787828976.834123:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787828976.834129:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828976.834133:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790603008:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828976.834141:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790603520/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828976.834151:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790603136:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828976.834155:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828976.834160:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828976.834163:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828976.834168:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5613 00000800:00000100:0.0:1787828976.834172:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828976.834175:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828976.834183:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828976.834184:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828976/real 1787828976] req@ffff9e6831aa8780 x1874668790603136/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828976.834190:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828976.834197:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790603136/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828976.834206:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790603648/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828976.834213:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828976.834220:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828976.834228:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790603136:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828976.834229:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828976.834235:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790603264:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828976.834236:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828976.834241:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828976.834247:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5613 00000100:00080000:1.0:1787828976.834252:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828976.834253:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828976.834260:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828976.834266:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828976.834274:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828976/real 1787828976] req@ffff9e6831aaa1c0 x1874668790603264/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828976.834283:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790603776/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828976.834285:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790603264/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828976.834312:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828976.834317:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828976.834317:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828976.834322:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790603264:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828976.834324:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828976.834328:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828976.834329:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790603392:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787828976.834335:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5613 00000100:00080000:1.0:1787828976.834340:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828976.834345:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787828976.834348:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1787828976.834353:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828976.834358:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828976/real 1787828976] req@ffff9e6831aa8000 x1874668790603392/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828976.834374:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790603392/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828976.834374:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790603904/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828976.834404:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828976.834409:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828976.834409:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828976.834416:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790603392:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787828976.834417:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828976.834421:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828976.834426:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5613 00000100:00080000:1.0:1787828976.834430:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828976.834431:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790603520:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787828976.834440:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787828976.834448:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1787828976.834448:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828976.834458:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828976/real 1787828976] req@ffff9e6831aaad00 x1874668790603520/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828976.834466:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790604032/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828976.834471:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790603520/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828976.834490:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828976.834495:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828976.834501:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790603520:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828976.834510:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790603648:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828976.834524:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828976.834532:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828976/real 1787828976] req@ffff9e6831aab480 x1874668790603648/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828976.834540:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790603648/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828976.834546:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828976.834548:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828976.834550:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790603648:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828976.834553:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790603776:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828976.834562:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828976.834568:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828976/real 1787828976] req@ffff9e6831aabc00 x1874668790603776/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828976.834575:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790603776/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828976.834580:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828976.834581:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828976.834583:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790603776:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828976.834586:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790603904:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828976.834596:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828976.834601:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828976/real 1787828976] req@ffff9e6831aa9a40 x1874668790603904/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828976.834608:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790603904/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828976.834612:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828976.834614:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828976.834616:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790603904:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828976.834625:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790604032:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828976.834634:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828976.834639:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828976/real 1787828976] req@ffff9e6831aaa940 x1874668790604032/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828976.834646:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790604032/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828976.834651:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828976.834653:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828976.834655:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790604032:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828977.345806:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828977.537926:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828978.369840:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828978.561856:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828979.393760:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828979.585889:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828980.417781:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828980.609844:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828981.441771:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828981.633849:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828981.953778:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828981.953799:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828981.953803:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828981.953812:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5618 00000100:00080000:1.0:1787828981.953821:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828981.953827:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828981.953857:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790604160/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828981.953877:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828981.953881:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828981.953882:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828981.953885:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5618 00000100:00080000:1.0:1787828981.953889:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828981.953892:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828981.953899:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828981.953911:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790604160:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828981.953913:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790604288/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828981.953923:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828981.953926:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828981.953927:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828981.953930:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5618 00000100:00080000:1.0:1787828981.953933:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828981.953936:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828981.953940:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828981.953948:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790604416/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828981.953954:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828981.953957:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828981.953959:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828981.953961:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5618 00000100:00080000:1.0:1787828981.953963:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828981.953965:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787828981.953970:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1787828981.953972:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828981.953982:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790604544/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828981.953989:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828981.953992:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828981.953993:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828981.953997:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5618 00000100:00080000:1.0:1787828981.954000:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828981.954002:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828981.954006:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828981.954008:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828981/real 1787828981] req@ffff9e6831aa8f00 x1874668790604160/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828981.954018:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790604672/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828981.954025:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828981.954027:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790604160/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828981.954029:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828981.954032:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828981.954039:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5618 00000100:00080000:1.0:1787828981.954044:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828981.954045:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828981.954049:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787828981.954054:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:1.0:1787828981.954054:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828981.954064:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790604160:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787828981.954067:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790604800/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828981.954078:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828981.954081:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828981.954082:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828981.954086:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5618 00000100:00080000:1.0:1787828981.954089:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828981.954092:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828981.954095:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828981.954100:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790604288:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828981.954106:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790604928/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828981.954115:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828981.954118:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828981.954120:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828981.954124:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5618 00000100:00080000:1.0:1787828981.954128:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828981.954133:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787828981.954134:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1787828981.954141:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828981.954151:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790605056/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828981.954154:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828981/real 1787828981] req@ffff9e6831aaa580 x1874668790604288/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828981.954169:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828981.954173:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828981.954174:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828981.954178:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790604288/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828981.954181:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5618 00000100:00080000:1.0:1787828981.954189:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828981.954191:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828981.954197:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787828981.954198:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828981.954204:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790604288:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787828981.954205:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828981.954210:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790604416:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787828981.954221:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790605184/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828981.954228:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828981.954240:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828981/real 1787828981] req@ffff9e6831aa9680 x1874668790604416/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828981.954249:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790604416/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828981.954256:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828981.954258:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828981.954260:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790604416:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828981.954272:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790604544:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828981.954288:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828981.954294:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828981/real 1787828981] req@ffff9e6831aa9e00 x1874668790604544/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828981.954302:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790604544/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828981.954310:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828981.954314:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828981.954317:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790604544:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828981.954324:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790604672:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828981.954346:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828981.954360:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828981/real 1787828981] req@ffff9e6831aaa940 x1874668790604672/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828981.954371:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790604672/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828981.954377:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828981.954380:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828981.954383:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790604672:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828981.954389:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790604800:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828981.954405:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828981.954416:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828981/real 1787828981] req@ffff9e6831aa9a40 x1874668790604800/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828981.954429:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790604800/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828981.954435:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828981.954438:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828981.954440:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790604800:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828981.954457:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790604928:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828981.954468:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828981.954474:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828981/real 1787828981] req@ffff9e6831aabc00 x1874668790604928/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828981.954481:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790604928/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828981.954491:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828981.954493:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828981.954495:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790604928:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828981.954498:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790605056:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828981.954507:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828981.954512:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828981/real 1787828981] req@ffff9e6831aab480 x1874668790605056/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828981.954519:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790605056/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828981.954524:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828981.954526:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828981.954528:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790605056:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828981.954531:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790605184:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828981.954540:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828981.954545:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828981/real 1787828981] req@ffff9e6831aaad00 x1874668790605184/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828981.954551:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790605184/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829006 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828981.954556:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828981.954558:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828981.954560:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790605184:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828982.465821:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828982.657907:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828983.489788:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828983.681861:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828984.513794:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828984.705912:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828985.537788:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828985.729885:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828986.561828:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828986.753884:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828987.073799:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828987.073808:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828987.073813:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828987.073821:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5623 00000100:00080000:1.0:1787828987.073830:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828987.073836:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787828987.073866:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790605312/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828987.073888:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828987.073891:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828987.073892:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828987.073896:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5623 00000100:00080000:1.0:1787828987.073898:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828987.073901:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828987.073909:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828987.073920:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790605440/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828987.073927:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828987.073929:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828987.073931:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828987.073935:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790605312:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787828987.073936:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5623 00000100:00080000:1.0:1787828987.073940:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828987.073945:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828987.073949:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828987.073957:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790605568/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828987.073963:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828987.073966:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828987.073967:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828987.073974:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5623 00000100:00080000:1.0:1787828987.073977:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828987.073979:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828987.073983:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828987.073990:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790605696/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828987.073996:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828987.073999:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828987.074000:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787828987.074005:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787828987.074006:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5623 00000100:00080000:1.0:1787828987.074012:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828987.074014:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828987.074017:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828987.074025:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790605824/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828987.074031:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828987.074033:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828987.074034:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828987.074037:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5623 00000100:00080000:1.0:1787828987.074039:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828987.074042:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828987.074046:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828987.074059:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790605952/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828987.074060:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828987/real 1787828987] req@ffff9e6831aa8000 x1874668790605312/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828987.074072:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828987.074078:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787828987.074081:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828987.074087:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5623 00000100:00080000:1.0:1787828987.074090:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828987.074093:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828987.074096:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828987.074101:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790605312/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828987.074105:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790606080/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828987.074132:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828987.074135:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828987.074135:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828987.074137:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828987.074145:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5623 00000100:00080000:0.0:1787828987.074146:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787828987.074148:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828987.074154:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828987.074158:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790605312:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787828987.074159:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828987.074168:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790606208/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828987.074175:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828987.074178:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828987.074181:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828987.074186:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5623 00000100:00100000:0.0:1787828987.074191:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790605440:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787828987.074191:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828987.074200:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787828987.074206:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787828987.074220:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790606336/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828987.074226:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828987.074246:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828987/real 1787828987] req@ffff9e6831aaa1c0 x1874668790605440/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828987.074262:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790605440/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828987.074273:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828987.074277:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828987.074281:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790605440:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828987.074288:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790605568:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828987.074310:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828987.074324:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828987/real 1787828987] req@ffff9e6831aa8780 x1874668790605568/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828987.074338:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790605568/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828987.074347:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828987.074351:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828987.074354:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790605568:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828987.074361:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790605696:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828987.074382:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828987.074393:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828987/real 1787828987] req@ffff9e6831aa8b40 x1874668790605696/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828987.074407:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790605696/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828987.074417:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828987.074420:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828987.074424:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790605696:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828987.074430:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790605824:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828987.074449:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828987.074461:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828987/real 1787828987] req@ffff9e6831aaad00 x1874668790605824/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828987.074474:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790605824/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828987.074484:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828987.074487:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828987.074491:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790605824:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828987.074498:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790605952:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828987.074517:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828987.074529:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828987/real 1787828987] req@ffff9e6831aab480 x1874668790605952/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828987.074553:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790605952/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828987.074564:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828987.074568:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828987.074571:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790605952:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828987.074578:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790606080:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828987.074599:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828987.074611:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828987/real 1787828987] req@ffff9e6831aabc00 x1874668790606080/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828987.074625:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790606080/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828987.074634:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828987.074638:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828987.074642:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790606080:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828987.074649:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790606208:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828987.074668:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828987.074679:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828987/real 1787828987] req@ffff9e6831aa9a40 x1874668790606208/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828987.074692:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790606208/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828987.074722:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828987.074727:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828987.074731:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790606208:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828987.074770:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790606336:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828987.074792:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828987.074804:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828987/real 1787828987] req@ffff9e6831aaa940 x1874668790606336/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828987.074818:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790606336/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828987.074827:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828987.074831:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828987.074835:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790606336:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828987.585814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787828987.777907:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828988.609801:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828988.801915:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828989.633835:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828989.825958:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828990.657793:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828990.849818:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828991.681819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828991.873861:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828992.193831:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828992.193843:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828992.193847:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828992.193857:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5628 00000100:00080000:2.0:1787828992.193866:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828992.193872:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828992.193918:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790606464/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828992.193942:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828992.193945:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828992.193946:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828992.193949:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5628 00000100:00080000:2.0:1787828992.193952:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828992.193954:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828992.193962:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828992.193972:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790606592/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828992.193980:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828992.193982:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828992.193984:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828992.193986:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5628 00000100:00080000:2.0:1787828992.193988:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828992.193990:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828992.193993:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828992.194001:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790606464:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787828992.194002:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790606720/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828992.194016:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828992.194018:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828992.194019:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828992.194021:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5628 00000100:00080000:2.0:1787828992.194023:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828992.194026:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828992.194029:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828992.194037:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790606848/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828992.194043:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828992.194045:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828992.194047:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828992.194050:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5628 00000100:00080000:2.0:1787828992.194053:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828992.194055:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828992.194058:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828992.194067:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790606976/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828992.194075:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828992.194076:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828992.194082:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828992.194083:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828992.194087:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5628 00000100:00080000:2.0:1787828992.194090:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828992.194093:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828992.194099:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828992.194123:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828992/real 1787828992] req@ffff9e6804420b40 x1874668790606464/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828992.194125:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790607104/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828992.194152:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828992.194155:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828992.194157:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828992.194159:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790606464/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828992.194163:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5628 00000100:00080000:2.0:1787828992.194170:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828992.194176:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787828992.194181:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787828992.194182:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787828992.194191:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828992.194197:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790607232/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828992.194200:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790606464:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828992.194214:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828992.194215:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790606592:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828992.194223:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828992.194227:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828992.194232:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5628 00000800:00000100:0.0:1787828992.194244:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787828992.194248:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828992.194256:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787828992.194258:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828992/real 1787828992] req@ffff9e68044230c0 x1874668790606592/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828992.194265:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828992.194271:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790606592/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828992.194279:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790607360/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828992.194280:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828992.194288:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828992.194295:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790606592:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787828992.194300:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828992.194304:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828992.194306:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828992.194310:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5628 00000100:00080000:2.0:1787828992.194312:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828992.194315:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828992.194319:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828992.194325:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790607488/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828992.194327:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790606720:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828992.194353:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828992.194363:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828992/real 1787828992] req@ffff9e68044212c0 x1874668790606720/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828992.194371:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790606720/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828992.194381:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828992.194384:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828992.194386:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790606720:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828992.194390:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790606848:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828992.194400:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828992.194406:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828992/real 1787828992] req@ffff9e6804421e00 x1874668790606848/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828992.194413:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790606848/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828992.194418:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828992.194420:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828992.194422:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790606848:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828992.194426:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790606976:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828992.194435:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828992.194441:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828992/real 1787828992] req@ffff9e6804423840 x1874668790606976/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828992.194448:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790606976/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828992.194453:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828992.194455:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828992.194458:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790606976:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828992.194461:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790607104:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828992.194471:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828992.194477:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828992/real 1787828992] req@ffff9e68044221c0 x1874668790607104/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828992.194484:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790607104/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828992.194490:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828992.194491:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828992.194493:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790607104:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828992.194496:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790607232:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828992.194506:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828992.194519:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828992/real 1787828992] req@ffff9e6804420780 x1874668790607232/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828992.194527:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790607232/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828992.194532:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828992.194534:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828992.194536:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790607232:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828992.194539:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790607360:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828992.194550:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828992.194556:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828992/real 1787828992] req@ffff9e6804422d00 x1874668790607360/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828992.194563:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790607360/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828992.194568:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828992.194570:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828992.194572:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790607360:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828992.194588:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790607488:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828992.194606:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828992.194618:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828992/real 1787828992] req@ffff9e6804420f00 x1874668790607488/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828992.194631:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790607488/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828992.194639:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828992.194642:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828992.194644:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790607488:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828992.705795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828992.897835:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828993.729800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828993.921810:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828994.753776:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828994.945831:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828995.777847:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828995.969887:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828996.801774:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828996.993827:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828997.313763:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828997.313771:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828997.313774:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828997.313783:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5634 00000100:00080000:2.0:1787828997.313791:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828997.313797:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787828997.313830:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790607616/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828997.313862:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828997.313866:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828997.313868:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828997.313873:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5634 00000100:00080000:2.0:1787828997.313877:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828997.313881:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828997.313891:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828997.313903:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790607744/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828997.313911:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828997.313914:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787828997.313915:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828997.313918:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5634 00000100:00080000:2.0:1787828997.313920:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828997.313925:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790607616:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828997.313925:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828997.313931:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828997.313939:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790607872/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828997.313950:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828997.313953:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828997.313954:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828997.313957:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5634 00000100:00080000:2.0:1787828997.313960:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828997.313962:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828997.313965:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787828997.313972:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790608000/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828997.313978:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828997.313981:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828997.313982:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828997.313984:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5634 00000100:00080000:2.0:1787828997.313986:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828997.313988:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828997.313991:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787828997.313996:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787828997.314001:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790608128/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828997.314007:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828997.314009:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828997.314011:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828997.314014:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5634 00000100:00080000:2.0:1787828997.314018:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828997.314021:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828997.314023:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828997.314030:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790608256/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828997.314036:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828997.314040:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787828997.314042:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828997/real 1787828997] req@ffff9e6804422940 x1874668790607616/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828997.314042:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828997.314048:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5634 00000100:00080000:2.0:1787828997.314052:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828997.314057:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828997.314062:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828997.314074:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790608384/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828997.314082:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790607616/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828997.314083:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828997.314088:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828997.314091:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828997.314098:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5634 00000100:00080000:2.0:1787828997.314102:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828997.314106:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787828997.314109:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787828997.314109:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787828997.314120:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787828997.314120:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790608512/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828997.314131:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790607616:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787828997.314137:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828997.314139:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828997.314140:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828997.314143:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5634 00000100:00080000:2.0:1787828997.314147:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828997.314151:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787828997.314155:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787828997.314159:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790607744:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787828997.314164:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790608640/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828997.314193:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828997.314208:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828997/real 1787828997] req@ffff9e6804423480 x1874668790607744/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828997.314222:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790607744/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828997.314232:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828997.314235:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828997.314239:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790607744:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828997.314261:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790607872:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828997.314282:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828997.314294:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828997/real 1787828997] req@ffff9e6804421680 x1874668790607872/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828997.314307:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790607872/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828997.314316:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828997.314320:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828997.314323:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790607872:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828997.314338:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790608000:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828997.314359:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828997.314371:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828997/real 1787828997] req@ffff9e6804422580 x1874668790608000/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828997.314385:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790608000/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828997.314393:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828997.314397:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828997.314400:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790608000:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828997.314406:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790608128:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828997.314424:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828997.314436:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828997/real 1787828997] req@ffff9e6804420f00 x1874668790608128/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828997.314449:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790608128/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828997.314458:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828997.314461:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828997.314464:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790608128:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828997.314470:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790608256:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828997.314488:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828997.314498:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828997/real 1787828997] req@ffff9e6804422d00 x1874668790608256/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828997.314512:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790608256/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828997.314523:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828997.314526:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828997.314530:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790608256:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828997.314536:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790608384:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787828997.314555:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828997.314564:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828997/real 1787828997] req@ffff9e6804420780 x1874668790608384/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828997.314576:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790608384/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828997.314585:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828997.314588:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828997.314592:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790608384:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787828997.314599:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790608512:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828997.314618:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828997.314628:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828997/real 1787828997] req@ffff9e68044221c0 x1874668790608512/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828997.314640:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790608512/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828997.314654:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828997.314658:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828997.314661:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790608512:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787828997.314668:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790608640:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787828997.314729:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787828997.314743:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828997/real 1787828997] req@ffff9e6804423840 x1874668790608640/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828997.314756:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790608640/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828997.314765:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828997.314769:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787828997.314772:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790608640:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787828997.825780:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828998.017900:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828998.849807:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787828999.041836:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828999.873804:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829000.066842:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829000.897790:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829001.089824:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829001.921784:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829002.113842:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829002.433804:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829002.433814:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829002.433819:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829002.433830:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5639 00000100:00080000:2.0:1787829002.433841:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829002.433849:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829002.433886:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790608768/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829002.433916:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829002.433920:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829002.433922:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829002.433927:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5639 00000100:00080000:2.0:1787829002.433931:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829002.433936:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829002.433947:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829002.433966:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790608768:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829002.433966:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790608896/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829002.433984:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829002.433988:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829002.433990:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829002.433995:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5639 00000100:00080000:2.0:1787829002.433999:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829002.434004:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829002.434010:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829002.434023:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829002.434026:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790609024/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829002.434038:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829002.434043:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829002.434045:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829002.434050:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5639 00000100:00080000:2.0:1787829002.434054:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829002.434060:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829002.434066:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829002.434067:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829002/real 1787829002] req@ffff9e6804421e00 x1874668790608768/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829002.434084:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790609152/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829002.434086:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790608768/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829002.434115:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829002.434117:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829002.434120:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829002.434127:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787829002.434128:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790608768:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829002.434132:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829002.434139:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5639 00000100:00080000:2.0:1787829002.434143:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829002.434149:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829002.434150:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790608896:192.168.203.109@tcp:38:kworker.0 00000008:00100000:2.0:1787829002.434158:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829002.434172:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829002.434179:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790609280/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829002.434184:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829002/real 1787829002] req@ffff9e68044212c0 x1874668790608896/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829002.434194:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790608896/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829002.434214:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829002.434218:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829002.434219:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829002.434222:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790608896:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829002.434226:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787829002.434231:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790609024:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829002.434232:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829002.434240:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5639 00000100:00080000:2.0:1787829002.434246:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829002.434247:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829002.434256:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829002.434259:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829002/real 1787829002] req@ffff9e68044230c0 x1874668790609024/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829002.434266:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829002.434270:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790609024/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829002.434276:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829002.434278:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829002.434284:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790609024:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829002.434285:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790609408/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829002.434289:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790609152:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787829002.434304:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829002.434309:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829002.434310:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829002.434314:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829002.434319:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829002/real 1787829002] req@ffff9e6804420b40 x1874668790609152/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829002.434323:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5639 00000100:00080000:2.0:1787829002.434330:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829002.434337:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829002.434343:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829002.434344:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790609152/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829002.434357:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787829002.434359:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790609536/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829002.434365:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829002.434375:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790609152:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787829002.434383:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829002.434388:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829002.434390:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829002.434394:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790609280:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787829002.434400:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5639 00000100:00080000:2.0:1787829002.434405:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829002.434411:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787829002.434417:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:2.0:1787829002.434417:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829002.434427:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829002/real 1787829002] req@ffff9e6804421e00 x1874668790609280/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829002.434435:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790609664/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829002.434441:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790609280/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829002.434461:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829002.434465:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829002.434466:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829002.434469:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790609280:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787829002.434474:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787829002.434478:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790609408:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829002.434480:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829002.434488:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5639 00000100:00080000:2.0:1787829002.434493:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829002.434494:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829002.434503:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829002.434510:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829002/real 1787829002] req@ffff9e6804423840 x1874668790609408/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829002.434511:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829002.434530:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790609792/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829002.434533:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790609408/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829002.434568:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829002.434572:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829002.434576:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790609408:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829002.434583:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790609536:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829002.434606:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829002.434618:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829002/real 1787829002] req@ffff9e68044221c0 x1874668790609536/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829002.434630:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790609536/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829002.434638:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829002.434656:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829002.434660:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790609536:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829002.434677:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790609664:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829002.434722:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829002.434732:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829002/real 1787829002] req@ffff9e6804420780 x1874668790609664/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829002.434740:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790609664/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829002.434746:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829002.434748:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829002.434750:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790609664:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829002.434755:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790609792:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829002.434767:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829002.434772:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829002/real 1787829002] req@ffff9e6804422d00 x1874668790609792/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829002.434779:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790609792/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829002.434784:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829002.434786:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829002.434788:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790609792:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829002.945840:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829003.137879:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829003.969791:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829004.161881:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829004.993797:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829005.185844:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829006.017807:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829006.209825:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829007.041786:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829007.233827:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829007.553808:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829007.553819:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829007.553823:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829007.553834:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5644 00000100:00080000:0.0:1787829007.553845:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829007.553853:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829007.553888:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790609920/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829007.553911:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829007.553914:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829007.553915:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829007.553918:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5644 00000100:00080000:0.0:1787829007.553921:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829007.553925:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829007.553934:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829007.553942:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790610048/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829007.553950:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829007.553954:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829007.553956:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829007.553958:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790609920:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829007.553960:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5644 00000100:00080000:0.0:1787829007.553965:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829007.553970:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829007.553973:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829007.553981:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790610176/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829007.553988:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829007.553990:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829007.553991:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829007.553994:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5644 00000100:00080000:0.0:1787829007.553996:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829007.553998:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829007.554001:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829007.554009:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790610304/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829007.554015:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829007.554018:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829007.554019:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829007.554021:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5644 00000100:00080000:0.0:1787829007.554023:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829007.554025:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829007.554028:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829007.554033:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829007.554040:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790610432/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829007.554047:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829007.554049:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829007.554051:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829007.554054:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5644 00000100:00080000:0.0:1787829007.554057:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829007.554060:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829007.554062:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829007.554069:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790610560/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829007.554075:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829007.554080:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829007.554081:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829007.554081:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829007/real 1787829007] req@ffff9e6803e99e00 x1874668790609920/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829007.554088:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5644 00000100:00080000:0.0:1787829007.554093:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829007.554099:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829007.554106:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829007.554115:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790610688/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829007.554120:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790609920/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829007.554123:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829007.554129:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829007.554131:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829007.554139:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5644 00000100:00080000:0.0:1787829007.554145:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829007.554148:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787829007.554149:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787829007.554151:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829007.554160:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829007.554162:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790610816/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829007.554170:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790609920:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829007.554172:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829007.554178:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829007.554181:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829007.554184:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5644 00000100:00080000:0.0:1787829007.554186:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829007.554189:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829007.554191:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829007.554197:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790610944/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829007.554208:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790610048:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829007.554239:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829007.554254:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829007/real 1787829007] req@ffff9e6803e99680 x1874668790610048/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829007.554268:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790610048/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829007.554278:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829007.554282:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829007.554285:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790610048:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829007.554292:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790610176:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829007.554317:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829007.554326:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829007/real 1787829007] req@ffff9e6803e9a580 x1874668790610176/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829007.554334:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790610176/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829007.554339:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829007.554341:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829007.554343:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790610176:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829007.554347:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790610304:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829007.554359:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829007.554365:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829007/real 1787829007] req@ffff9e6803e9b480 x1874668790610304/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829007.554371:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790610304/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829007.554377:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829007.554379:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829007.554380:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790610304:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829007.554384:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790610432:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829007.554393:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829007.554399:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829007/real 1787829007] req@ffff9e6803e9b840 x1874668790610432/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829007.554411:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790610432/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829007.554416:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829007.554418:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829007.554420:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790610432:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829007.554423:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790610560:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829007.554433:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829007.554438:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829007/real 1787829007] req@ffff9e6803e9a1c0 x1874668790610560/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829007.554445:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790610560/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829007.554451:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829007.554453:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829007.554455:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790610560:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829007.554458:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790610688:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829007.554467:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829007.554472:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829007/real 1787829007] req@ffff9e6803e98780 x1874668790610688/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829007.554479:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790610688/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829007.554488:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829007.554491:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829007.554494:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790610688:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829007.554500:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790610816:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829007.554526:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829007.554538:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829007/real 1787829007] req@ffff9e6803e98b40 x1874668790610816/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829007.554549:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790610816/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829007.554555:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829007.554557:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829007.554560:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790610816:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829007.554575:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790610944:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829007.554590:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829007.554596:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829007/real 1787829007] req@ffff9e6803e9bc00 x1874668790610944/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829007.554617:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790610944/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829007.554623:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829007.554626:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829007.554628:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790610944:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829008.065789:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829008.257903:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829009.089752:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829009.281842:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829010.113756:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829010.305844:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829011.137784:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829011.329843:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829012.161787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829012.353943:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829012.673820:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829012.673831:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829012.673835:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829012.673846:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5649 00000100:00080000:2.0:1787829012.673858:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829012.673865:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829012.673913:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790611072/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829012.673941:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829012.673944:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829012.673946:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829012.673950:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5649 00000100:00080000:2.0:1787829012.673954:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829012.673959:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829012.673975:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829012.673987:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790611072:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829012.673987:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790611200/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829012.674018:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829012.674022:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829012.674024:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829012.674028:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5649 00000100:00080000:2.0:1787829012.674032:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829012.674035:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829012.674039:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829012.674051:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790611328/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829012.674064:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829012.674068:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829012.674069:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829012.674072:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5649 00000100:00080000:2.0:1787829012.674074:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829012.674079:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829012.674085:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829012.674090:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829012.674100:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790611456/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829012.674107:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829012.674110:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829012.674112:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829012.674116:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5649 00000100:00080000:2.0:1787829012.674121:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829012.674125:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829012.674126:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829012/real 1787829012] req@ffff9e6804420f00 x1874668790611072/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829012.674131:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829012.674149:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790611584/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829012.674158:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790611072/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829012.674170:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829012.674186:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829012.674191:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829012.674199:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5649 00000100:00080000:2.0:1787829012.674202:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829012.674204:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829012.674205:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787829012.674213:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:2.0:1787829012.674213:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829012.674226:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790611072:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829012.674236:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790611712/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829012.674243:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829012.674246:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829012.674248:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829012.674252:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5649 00000100:00080000:2.0:1787829012.674256:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829012.674260:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790611200:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829012.674260:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829012.674273:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829012.674289:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790611840/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829012.674298:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829012.674329:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829012.674331:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829012/real 1787829012] req@ffff9e6804422580 x1874668790611200/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829012.674337:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829012.674342:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829012.674360:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5649 00000100:00080000:2.0:1787829012.674366:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829012.674367:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790611200/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829012.674378:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829012.674395:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829012.674397:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829012.674403:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829012.674407:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790611200:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829012.674412:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790611968/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829012.674420:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790611328:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829012.674445:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829012.674461:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829012.674466:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829012/real 1787829012] req@ffff9e6804421680 x1874668790611328/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829012.674479:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829012.674487:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829012.674488:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790611328/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829012.674505:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829012.674510:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5649 00000100:00080000:1.0:1787829012.674515:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829012.674521:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829012.674525:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790611328:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829012.674534:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829012.674541:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790611456:192.168.203.109@tcp:8:kworker.0 00000008:00100000:2.0:1787829012.674547:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829012.674563:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829012.674588:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790612096/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829012.674590:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829012/real 1787829012] req@ffff9e6804423480 x1874668790611456/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829012.674628:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790611456/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829012.674638:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829012.674641:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829012.674644:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790611456:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829012.674650:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790611584:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829012.674668:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829012.674674:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829012/real 1787829012] req@ffff9e6804422940 x1874668790611584/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829012.674681:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790611584/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829012.674687:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829012.674689:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829012.674691:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790611584:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829012.674713:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790611712:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829012.674726:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829012.674733:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829012/real 1787829012] req@ffff9e6804422d00 x1874668790611712/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829012.674740:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790611712/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829012.674747:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829012.674749:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829012.674751:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790611712:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829012.674766:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790611840:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829012.674777:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829012.674783:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829012/real 1787829012] req@ffff9e6804420780 x1874668790611840/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829012.674791:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790611840/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829012.674795:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829012.674797:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829012.674800:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790611840:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829012.674804:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790611968:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829012.674814:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829012.674819:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829012/real 1787829012] req@ffff9e68044221c0 x1874668790611968/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829012.674826:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790611968/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829012.674830:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829012.674832:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829012.674834:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790611968:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829012.674837:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790612096:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829012.674847:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829012.674852:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829012/real 1787829012] req@ffff9e6804423840 x1874668790612096/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829012.674859:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790612096/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829012.674863:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829012.674865:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829012.674867:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790612096:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829013.185800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829013.377894:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829014.209791:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829014.401916:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829015.233819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829015.425977:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829016.257822:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829016.449903:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829017.281795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829017.473910:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829017.793831:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829017.793857:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829017.793863:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829017.793874:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5654 00000100:00080000:1.0:1787829017.793892:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829017.793900:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829017.793953:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790612224/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829017.793975:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829017.793978:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829017.793985:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829017.793990:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5654 00000100:00080000:1.0:1787829017.793993:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829017.794000:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829017.794008:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790612224:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787829017.794010:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829017.794032:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790612352/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829017.794043:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829017.794047:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829017.794049:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829017.794065:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5654 00000100:00080000:1.0:1787829017.794069:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829017.794072:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829017.794077:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829017.794086:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790612480/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829017.794100:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829017.794104:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1787829017.794110:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829017.794111:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829017.794121:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5654 00000100:00080000:1.0:1787829017.794127:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829017.794134:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829017.794150:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829017.794158:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829017/real 1787829017] req@ffff9e6831aa9e00 x1874668790612224/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829017.794172:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790612608/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829017.794178:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790612224/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829017.794194:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829017.794201:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829017.794203:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829017.794205:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829017.794211:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829017.794216:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5654 00000100:00100000:0.0:1787829017.794221:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790612224:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829017.794229:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829017.794254:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829017.794265:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829017.794268:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790612352:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829017.794301:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829017.794316:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790612736/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829017.794330:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829017/real 1787829017] req@ffff9e6831aa9680 x1874668790612352/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829017.794345:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829017.794351:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829017.794353:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829017.794361:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5654 00000100:00100000:0.0:1787829017.794364:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790612352/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829017.794384:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829017.794394:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787829017.794395:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829017.794403:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:1.0:1787829017.794404:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829017.794416:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790612352:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829017.794427:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790612864/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829017.794430:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790612480:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829017.794474:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829017.794478:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829017.794480:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829017.794485:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829017.794490:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5654 00000100:00080000:1.0:1787829017.794498:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829017.794506:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829017.794510:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829017/real 1787829017] req@ffff9e6831aaa580 x1874668790612480/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829017.794513:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829017.794560:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790612480/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829017.794560:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790612992/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829017.794633:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829017.794646:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829017.794652:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829017.794663:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5654 00000100:00080000:0.0:1787829017.794665:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829017.794667:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829017.794676:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829017.794677:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829017.794703:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790612480:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829017.794775:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790612608:192.168.203.109@tcp:8:kworker.0 00000008:00100000:1.0:1787829017.794775:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829017.794803:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790613120/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829017.794828:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829017.794835:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829017.794841:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829017.794845:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829017.794851:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829017/real 1787829017] req@ffff9e6831aa8f00 x1874668790612608/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829017.794852:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5654 00000100:00080000:1.0:1787829017.794885:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829017.794887:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790612608/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829017.794894:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829017.794911:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829017.794912:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829017.794924:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829017.794928:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790612608:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829017.794935:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790613248/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829017.794957:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790612736:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829017.794976:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829017.794983:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829017/real 1787829017] req@ffff9e6831aa9e00 x1874668790612736/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829017.794991:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790612736/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829017.794996:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829017.794998:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829017.795001:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790612736:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829017.795006:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790612864:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829017.795017:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829017.795023:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829017/real 1787829017] req@ffff9e6831aaa940 x1874668790612864/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829017.795030:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790612864/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829017.795037:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829017.795039:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829017.795041:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790612864:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829017.795044:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790612992:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829017.795055:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829017.795061:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829017/real 1787829017] req@ffff9e6831aa9a40 x1874668790612992/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829017.795069:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790612992/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829017.795074:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829017.795076:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829017.795078:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790612992:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829017.795081:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790613120:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829017.795092:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829017.795099:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829017/real 1787829017] req@ffff9e6831aabc00 x1874668790613120/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829017.795107:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790613120/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829017.795112:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829017.795114:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829017.795116:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790613120:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829017.795119:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790613248:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829017.795130:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829017.795136:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829017/real 1787829017] req@ffff9e6831aab480 x1874668790613248/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829017.795143:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790613248/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829017.795148:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829017.795150:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829017.795152:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790613248:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829018.305857:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829018.497983:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829019.329818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829019.521884:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829020.353762:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829020.545884:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829021.377787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829021.569857:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829022.401820:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829022.593898:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829022.913797:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829022.913823:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829022.913828:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829022.913840:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5659 00000100:00080000:2.0:1787829022.913852:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829022.913861:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829022.913901:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790613376/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829022.913945:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829022.913949:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829022.913950:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829022.913954:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5659 00000100:00080000:2.0:1787829022.913957:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829022.913961:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829022.913971:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829022.914000:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790613504/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829022.914001:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790613376:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829022.914016:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829022.914018:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829022.914019:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829022.914021:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5659 00000100:00080000:2.0:1787829022.914023:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829022.914026:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829022.914029:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829022.914038:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790613632/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829022.914044:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829022.914046:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829022.914047:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829022.914050:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5659 00000100:00080000:2.0:1787829022.914070:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829022.914073:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829022.914077:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829022.914082:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829022.914086:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790613760/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829022.914093:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829022.914096:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829022.914097:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829022.914100:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5659 00000100:00080000:2.0:1787829022.914102:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829022.914105:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829022.914108:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829022.914119:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829022/real 1787829022] req@ffff9e6804421e00 x1874668790613376/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829022.914119:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790613888/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829022.914145:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829022.914148:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829022.914150:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829022.914154:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5659 00000100:00100000:0.0:1787829022.914155:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790613376/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829022.914157:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829022.914163:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829022.914169:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829022.914174:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787829022.914178:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790614016/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829022.914180:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829022.914188:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790613376:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829022.914194:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829022.914197:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829022.914198:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829022.914202:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5659 00000100:00080000:2.0:1787829022.914204:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829022.914206:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829022.914208:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829022.914212:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790613504:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829022.914218:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790614144/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829022.914226:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829022.914228:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829022.914230:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829022.914233:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829022.914234:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5659 00000100:00080000:2.0:1787829022.914240:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829022.914244:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829022.914248:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829022.914251:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829022/real 1787829022] req@ffff9e6804420b40 x1874668790613504/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829022.914258:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790614272/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829022.914266:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790613504/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829022.914268:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829022.914274:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829022.914277:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829022.914285:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5659 00000100:00080000:0.0:1787829022.914286:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829022.914287:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829022.914294:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829022.914295:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829022.914301:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829022.914302:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790613504:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829022.914311:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790614400/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829022.914312:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790613632:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829022.914334:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829022.914343:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829022/real 1787829022] req@ffff9e68044230c0 x1874668790613632/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829022.914350:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790613632/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829022.914355:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829022.914357:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829022.914359:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790613632:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829022.914363:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790613760:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829022.914373:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829022.914379:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829022/real 1787829022] req@ffff9e68044212c0 x1874668790613760/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829022.914386:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790613760/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829022.914391:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829022.914393:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829022.914394:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790613760:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829022.914398:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790613888:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829022.914407:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829022.914412:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829022/real 1787829022] req@ffff9e6804423840 x1874668790613888/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829022.914418:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790613888/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829022.914423:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829022.914425:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829022.914427:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790613888:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829022.914430:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790614016:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829022.914439:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829022.914445:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829022/real 1787829022] req@ffff9e68044221c0 x1874668790614016/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829022.914452:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790614016/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829022.914459:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829022.914460:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829022.914462:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790614016:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829022.914473:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790614144:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829022.914482:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829022.914488:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829022/real 1787829022] req@ffff9e6804420780 x1874668790614144/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829022.914495:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790614144/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829022.914515:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829022.914517:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829022.914519:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790614144:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829022.914523:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790614272:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829022.914534:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829022.914541:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829022/real 1787829022] req@ffff9e6804422d00 x1874668790614272/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829022.914557:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790614272/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829022.914567:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829022.914569:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829022.914571:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790614272:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829022.914574:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790614400:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829022.914584:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829022.914589:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829022/real 1787829022] req@ffff9e6804422940 x1874668790614400/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829022.914596:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790614400/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829047 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829022.914601:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829022.914603:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829022.914605:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790614400:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829023.425789:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829023.617844:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829024.449840:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829024.641835:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829025.473776:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829025.665870:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829026.497795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829026.689825:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829027.521770:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829027.713846:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829028.033781:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829028.033792:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829028.033796:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829028.033807:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5664 00000100:00080000:2.0:1787829028.033819:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829028.033826:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829028.033862:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790614528/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829028.033887:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829028.033891:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829028.033893:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829028.033897:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5664 00000100:00080000:2.0:1787829028.033900:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829028.033904:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829028.033913:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829028.033931:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790614528:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829028.033932:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790614656/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829028.033950:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829028.033953:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829028.033954:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829028.033958:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5664 00000100:00080000:2.0:1787829028.033962:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829028.033965:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829028.033969:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829028.033979:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790614784/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829028.033994:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829028.033997:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829028.033998:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829028.034003:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5664 00000800:00000100:0.0:1787829028.034008:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829028.034011:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829028.034020:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829028.034027:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829028.034045:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790614912/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829028.034060:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829028/real 1787829028] req@ffff9e6804423480 x1874668790614528/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829028.034063:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829028.034074:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829028.034080:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829028.034093:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5664 00000100:00080000:2.0:1787829028.034098:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829028.034105:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790614528/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829028.034107:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829028.034116:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829028.034124:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829028.034132:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829028.034136:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790615040/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829028.034141:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790614528:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829028.034158:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829028.034162:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829028.034164:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829028.034168:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5664 00000100:00080000:2.0:1787829028.034172:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829028.034176:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829028.034180:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829028.034188:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790615168/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829028.034192:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790614656:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829028.034210:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829028.034214:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829028.034216:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829028.034221:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5664 00000100:00080000:2.0:1787829028.034226:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829028.034230:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787829028.034235:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:2.0:1787829028.034239:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829028.034254:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790615296/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829028.034256:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829028/real 1787829028] req@ffff9e6804421680 x1874668790614656/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829028.034283:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829028.034287:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787829028.034291:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790614656/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829028.034291:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829028.034302:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5664 00000100:00080000:2.0:1787829028.034310:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829028.034314:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829028.034315:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829028.034326:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829028.034327:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829028.034334:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790614656:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829028.034340:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790614784:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829028.034362:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790615424/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829028.034368:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829028.034382:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829028/real 1787829028] req@ffff9e6804422580 x1874668790614784/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829028.034391:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829028.034401:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829028.034404:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829028.034408:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790614784/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829028.034413:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5664 00000100:00080000:2.0:1787829028.034420:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829028.034426:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829028.034427:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787829028.034435:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:2.0:1787829028.034437:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829028.034441:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790614784:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829028.034451:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790614912:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829028.034455:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790615552/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829028.034499:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829028.034513:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829028/real 1787829028] req@ffff9e6804420f00 x1874668790614912/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829028.034524:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790614912/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829028.034532:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829028.034535:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829028.034537:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790614912:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829028.034542:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790615040:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829028.034557:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829028.034565:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829028/real 1787829028] req@ffff9e6804422940 x1874668790615040/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829028.034574:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790615040/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829028.034581:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829028.034583:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829028.034585:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790615040:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829028.034600:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790615168:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829028.034613:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829028.034621:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829028/real 1787829028] req@ffff9e6804422d00 x1874668790615168/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829028.034631:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790615168/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829028.034645:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829028.034648:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829028.034650:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790615168:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829028.034655:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790615296:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829028.034667:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829028.034675:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829028/real 1787829028] req@ffff9e6804420780 x1874668790615296/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829028.034684:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790615296/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829028.034690:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829028.034693:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829028.034718:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790615296:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829028.034723:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790615424:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829028.034741:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829028.034756:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829028/real 1787829028] req@ffff9e68044221c0 x1874668790615424/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829028.034773:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790615424/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829028.034784:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829028.034788:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829028.034794:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790615424:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829028.034802:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790615552:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829028.034826:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829028.034836:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829028/real 1787829028] req@ffff9e6804423840 x1874668790615552/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829028.034846:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790615552/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829028.034853:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829028.034856:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829028.034858:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790615552:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829028.545822:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829028.737868:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829029.569781:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829029.761886:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829030.593838:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829030.785857:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829031.617798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829031.809859:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829032.641820:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829032.833886:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829033.153823:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829033.153832:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829033.153836:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829033.153845:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5669 00000100:00080000:1.0:1787829033.153855:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829033.153862:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829033.153896:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790615680/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829033.153921:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829033.153924:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829033.153925:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829033.153928:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5669 00000100:00080000:1.0:1787829033.153931:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829033.153934:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829033.153943:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829033.153959:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790615808/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829033.153973:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829033.153979:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829033.153984:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829033.153987:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790615680:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829033.153992:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5669 00000100:00080000:1.0:1787829033.154001:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829033.154006:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829033.154011:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829033.154023:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790615936/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829033.154031:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829033.154034:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829033.154035:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829033.154038:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5669 00000100:00080000:1.0:1787829033.154042:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829033.154045:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829033.154049:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829033.154056:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790616064/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829033.154063:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829033.154065:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829033.154067:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829033.154069:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5669 00000100:00080000:1.0:1787829033.154081:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829033.154084:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829033.154087:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829033.154096:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790616192/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829033.154104:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829033.154107:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829033.154108:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829033.154111:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5669 00000100:00080000:1.0:1787829033.154115:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829033.154118:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829033.154124:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829033.154136:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790616320/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829033.154142:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829033.154145:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829033.154148:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829033.154154:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790615808:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829033.154165:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5669 00000100:00080000:1.0:1787829033.154183:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829033.154193:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790615936:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829033.154194:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829033.154215:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829033.154249:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790616064:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829033.154251:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790616448/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829033.154272:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829033.154275:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829033.154277:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829033.154287:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790616192:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787829033.154287:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5669 00000100:00080000:1.0:1787829033.154301:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829033.154307:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829033.154312:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829033.154319:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790616320:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829033.154325:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790616576/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829033.154335:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829033.154339:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829033.154341:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829033.154346:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5669 00000100:00100000:0.0:1787829033.154352:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790616448:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829033.154354:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829033.154365:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829033.154372:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829033.154386:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790616576:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829033.154389:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790616704/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829033.154415:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790616704:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829033.665758:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829033.857874:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829034.689833:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829034.881838:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829035.713773:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829035.905907:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787829036.225930:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:2.0:1787829036.225977:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787829036.225991:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787829036.226029:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:2.0:1787829036.226042:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787829036.226052:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1787829036.226063:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829033/real 1787829036] req@ffff9e6831aaa940 x1874668790616704/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787829036.226066:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787829036.226077:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1787829036.226093:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790616704/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787829036.226095:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787829036.226106:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787829036.226117:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00080000:0.0:1787829036.226120:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829036.226128:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829036.226133:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790616704:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829036.226142:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829033/real 1787829036] req@ffff9e6831aabc00 x1874668790616448/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829036.226157:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790616448/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829036.226166:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829036.226169:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829036.226172:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790616448:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829036.226179:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829033/real 1787829036] req@ffff9e6831aa9a40 x1874668790616576/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829036.226190:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790616576/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829036.226198:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829036.226202:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829036.226205:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790616576:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829036.226213:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829033/real 1787829036] req@ffff9e6831aa8b40 x1874668790615808/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829036.226224:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790615808/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829036.226232:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829036.226235:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829036.226238:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790615808:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829036.226244:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829033/real 1787829036] req@ffff9e6831aa8780 x1874668790615936/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829036.226254:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790615936/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829036.226268:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829036.226272:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829036.226276:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790615936:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829036.226282:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829033/real 1787829036] req@ffff9e6831aaa1c0 x1874668790616064/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829036.226294:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790616064/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829036.226303:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829036.226306:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829036.226309:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790616064:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829036.226317:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829033/real 1787829036] req@ffff9e6831aa8000 x1874668790616192/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829036.226329:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790616192/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829036.226337:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829036.226340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829036.226343:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790616192:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829036.226349:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829033/real 1787829036] req@ffff9e6831aab480 x1874668790616320/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829036.226359:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790616320/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829036.226368:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829036.226372:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829036.226375:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790616320:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829036.226382:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829033/real 1787829036] req@ffff9e6831aaad00 x1874668790615680/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829036.226395:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790615680/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829036.226403:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829036.226422:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829036.226426:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790615680:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829036.737802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829036.929817:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829037.761762:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829037.953858:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829038.273789:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829038.273800:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829038.273805:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829038.273815:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5674 00000100:00080000:1.0:1787829038.273827:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829038.273834:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829038.273866:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790616832/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829038.273904:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829038.273908:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829038.273910:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829038.273915:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5674 00000100:00080000:1.0:1787829038.273919:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829038.273923:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829038.273933:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829038.273950:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790616960/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829038.273962:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829038.273965:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790616832:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829038.273967:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829038.273971:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829038.274002:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5674 00000100:00080000:1.0:1787829038.274007:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829038.274012:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829038.274018:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829038.274034:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790617088/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829038.274044:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829038.274058:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829038.274062:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829038.274064:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829038.274070:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5674 00000100:00080000:1.0:1787829038.274074:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829038.274079:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829038.274086:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829038.274093:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829038/real 1787829038] req@ffff9e6831aa9e00 x1874668790616832/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829038.274101:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790617216/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829038.274126:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829038.274132:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829038.274134:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829038.274143:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790616832/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829038.274155:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5674 00000100:00080000:2.0:1787829038.274156:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829038.274163:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829038.274165:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829038.274173:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829038.274174:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790616832:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787829038.274183:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829038.274201:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790617344/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829038.274207:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790616960:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829038.274219:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829038.274224:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829038.274226:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829038.274232:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5674 00000800:00000100:2.0:1787829038.274240:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829038.274244:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829038.274251:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829038.274256:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829038.274263:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829038/real 1787829038] req@ffff9e6831aa8f00 x1874668790616960/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829038.274271:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790617472/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829038.274284:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790616960/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829038.274285:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829038.274294:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829038.274298:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829038.274306:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5674 00000100:00080000:2.0:1787829038.274307:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829038.274311:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829038.274314:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829038.274319:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829038.274322:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790616960:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787829038.274326:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829038.274338:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790617088:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829038.274345:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790617600/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829038.274356:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829038.274360:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829038.274362:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829038.274367:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829038.274370:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5674 00000100:00080000:1.0:1787829038.274377:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829038.274381:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829038.274386:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829038/real 1787829038] req@ffff9e6831aaa580 x1874668790617088/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829038.274387:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829038.274419:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790617088/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829038.274425:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790617728/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829038.274454:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829038.274463:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829038.274467:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829038.274470:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790617088:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829038.274474:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829038.274478:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829038.274482:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790617216:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787829038.274486:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5674 00000100:00080000:1.0:1787829038.274492:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829038.274497:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829038.274502:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829038.274514:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829038.274518:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790617856/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829038.274532:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829038/real 1787829038] req@ffff9e6831aa9680 x1874668790617216/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829038.274546:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790617216/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829038.274555:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829038.274559:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829038.274562:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790617216:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829038.274578:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790617344:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829038.274598:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829038.274616:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829038/real 1787829038] req@ffff9e6831aaad00 x1874668790617344/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829038.274628:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790617344/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829038.274636:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829038.274640:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829038.274643:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790617344:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829038.274649:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790617472:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829038.274669:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829038.274681:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829038/real 1787829038] req@ffff9e6831aab480 x1874668790617472/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829038.274693:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790617472/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829038.274726:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829038.274730:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829038.274733:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790617472:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829038.274740:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790617600:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829038.274764:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829038.274779:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829038/real 1787829038] req@ffff9e6831aa8000 x1874668790617600/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829038.274794:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790617600/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829038.274802:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829038.274806:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829038.274809:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790617600:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829038.274815:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790617728:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829038.274836:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829038.274849:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829038/real 1787829038] req@ffff9e6831aaa1c0 x1874668790617728/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829038.274860:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790617728/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829038.274868:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829038.274871:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829038.274874:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790617728:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829038.274879:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790617856:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829038.274899:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829038.274910:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829038/real 1787829038] req@ffff9e6831aa8780 x1874668790617856/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829038.274922:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790617856/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829038.274930:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829038.274932:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829038.274935:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790617856:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829038.785774:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829038.977833:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829039.809815:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829040.001862:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829040.833787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829041.025823:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829041.857796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829042.049833:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829042.881782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829043.073828:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829043.393792:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829043.393801:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829043.393804:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829043.393813:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5680 00000100:00080000:0.0:1787829043.393823:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829043.393829:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829043.393861:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790617984/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829043.393881:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829043.393884:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829043.393886:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829043.393889:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5680 00000100:00080000:0.0:1787829043.393891:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829043.393893:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829043.393901:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829043.393910:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790618112/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829043.393918:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829043.393921:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829043.393922:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829043.393927:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790617984:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829043.393928:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5680 00000100:00080000:0.0:1787829043.393933:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829043.393938:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829043.393942:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829043.393951:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790618240/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829043.393958:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829043.393960:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829043.393962:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829043.393964:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5680 00000100:00080000:0.0:1787829043.393967:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829043.393970:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829043.393973:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829043.393982:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790618368/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829043.393989:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829043.393995:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829043.393998:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829043.393999:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829043.394002:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5680 00000100:00080000:0.0:1787829043.394005:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829043.394007:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829043.394011:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829043.394019:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790618496/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829043.394025:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829043.394027:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829043.394029:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829043.394034:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829043/real 1787829043] req@ffff9e6803e9a940 x1874668790617984/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829043.394035:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5680 00000100:00080000:0.0:1787829043.394041:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829043.394048:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829043.394053:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829043.394060:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790617984/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829043.394063:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790618624/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829043.394077:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829043.394085:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829043.394089:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829043.394092:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790617984:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829043.394094:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829043.394098:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829043.394102:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5680 00000100:00080000:0.0:1787829043.394104:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829043.394107:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829043.394109:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829043.394119:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790618112:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829043.394120:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790618752/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829043.394134:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829043.394137:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829043.394138:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829043.394141:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5680 00000100:00080000:0.0:1787829043.394144:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829043.394148:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829043.394149:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829043.394155:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829043.394164:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829043/real 1787829043] req@ffff9e6803e9ad00 x1874668790618112/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829043.394167:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790618880/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829043.394188:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829043.394190:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790618112/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829043.394191:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829043.394195:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829043.394202:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5680 00000100:00080000:0.0:1787829043.394208:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829043.394208:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829043.394212:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787829043.394216:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:0.0:1787829043.394218:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829043.394220:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790618112:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829043.394230:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790619008/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829043.394231:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790618240:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829043.394251:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829043.394260:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829043/real 1787829043] req@ffff9e6803e98f00 x1874668790618240/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829043.394269:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790618240/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829043.394274:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829043.394276:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829043.394278:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790618240:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829043.394282:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790618368:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829043.394292:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829043.394298:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829043/real 1787829043] req@ffff9e6803e992c0 x1874668790618368/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829043.394305:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790618368/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829043.394311:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829043.394312:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829043.394314:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790618368:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829043.394317:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790618496:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829043.394326:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829043.394332:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829043/real 1787829043] req@ffff9e6803e9bc00 x1874668790618496/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829043.394339:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790618496/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829043.394343:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829043.394345:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829043.394347:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790618496:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829043.394350:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790618624:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829043.394378:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829043.394387:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829043/real 1787829043] req@ffff9e6803e98b40 x1874668790618624/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829043.394395:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790618624/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829043.394401:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829043.394403:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829043.394405:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790618624:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829043.394418:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790618752:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829043.394429:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829043.394436:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829043/real 1787829043] req@ffff9e6803e98780 x1874668790618752/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829043.394443:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790618752/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829043.394448:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829043.394449:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829043.394451:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790618752:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829043.394455:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790618880:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829043.394464:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829043.394469:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829043/real 1787829043] req@ffff9e6803e9a1c0 x1874668790618880/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829043.394476:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790618880/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829043.394486:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829043.394487:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829043.394489:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790618880:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829043.394493:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790619008:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829043.394502:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829043.394507:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829043/real 1787829043] req@ffff9e6803e9b840 x1874668790619008/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829043.394513:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790619008/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829043.394518:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829043.394520:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829043.394522:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790619008:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829043.905803:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829044.097866:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829044.929778:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829045.121853:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829045.953834:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829046.145866:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829046.977841:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829047.169884:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829048.001778:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829048.193908:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829048.513860:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829048.513874:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829048.513880:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829048.513892:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5685 00000100:00080000:0.0:1787829048.513908:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829048.513917:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829048.513969:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790619136/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829048.514004:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829048.514068:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790619136:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829048.514077:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829048.514082:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829048.514089:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5685 00000100:00080000:0.0:1787829048.514097:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829048.514104:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829048.514125:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829048.514147:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790619264/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829048.514149:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829048.514170:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829048.514175:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829048.514178:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829048.514183:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5685 00000100:00080000:0.0:1787829048.514189:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829048.514195:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829048.514201:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829048.514205:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829048/real 1787829048] req@ffff9e6803e9b480 x1874668790619136/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829048.514220:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790619392/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829048.514233:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790619136/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829048.514238:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829048.514251:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829048.514254:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829048.514262:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829048.514263:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5685 00000100:00080000:0.0:1787829048.514276:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829048.514277:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829048.514289:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829048.514290:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790619136:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1787829048.514300:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829048.514333:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790619520/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829048.514345:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790619264:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829048.514353:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829048.514359:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829048.514362:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829048.514368:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5685 00000100:00080000:0.0:1787829048.514376:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829048.514382:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829048.514387:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829048.514396:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829048.514408:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829048/real 1787829048] req@ffff9e6803e9a580 x1874668790619264/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829048.514417:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790619648/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829048.514435:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790619264/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829048.514437:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829048.514451:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829048.514455:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829048.514468:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5685 00000100:00080000:2.0:1787829048.514469:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829048.514478:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829048.514481:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829048.514488:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829048.514490:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790619264:192.168.203.109@tcp:38:kworker.0 00000008:00100000:0.0:1787829048.514500:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829048.514507:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790619392:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829048.514521:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790619776/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829048.514534:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829048.514543:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829048.514545:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829048.514549:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829048.514558:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5685 00000100:00080000:0.0:1787829048.514564:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829048.514566:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829048/real 1787829048] req@ffff9e6803e99680 x1874668790619392/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829048.514583:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829048.514588:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790619392/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829048.514594:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829048.514609:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829048.514616:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829048.514619:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790619904/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829048.514626:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790619392:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829048.514651:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829048.514656:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829048.514660:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829048.514662:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790619520:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787829048.514670:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5685 00000100:00080000:0.0:1787829048.514677:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829048.514683:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787829048.514688:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787829048.514690:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829048.514733:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829048/real 1787829048] req@ffff9e6803e99e00 x1874668790619520/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829048.514754:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790619520/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829048.514762:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790620032/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829048.514764:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829048.514771:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829048.514778:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790619520:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829048.514794:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790619648:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787829048.514809:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829048.514814:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1787829048.514819:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829048.514821:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829048.514832:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829048/real 1787829048] req@ffff9e6803e9b480 x1874668790619648/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829048.514833:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5685 00000100:00080000:0.0:1787829048.514848:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829048.514855:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790619648/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829048.514863:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787829048.514870:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829048.514875:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:0.0:1787829048.514878:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829048.514878:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790619648:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829048.514892:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790619776:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829048.514910:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790620160/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829048.514916:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829048.514928:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829048/real 1787829048] req@ffff9e6803e9b840 x1874668790619776/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829048.514951:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790619776/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829048.514964:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829048.514968:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829048.514972:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790619776:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829048.514980:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790619904:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829048.515002:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829048.515015:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829048/real 1787829048] req@ffff9e6803e9a1c0 x1874668790619904/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829048.515028:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790619904/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829048.515037:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829048.515041:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829048.515044:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790619904:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829048.515062:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790620032:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829048.515079:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829048.515087:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829048/real 1787829048] req@ffff9e6803e98780 x1874668790620032/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829048.515095:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790620032/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829048.515100:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829048.515102:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829048.515104:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790620032:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829048.515109:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790620160:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829048.515119:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829048.515124:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829048/real 1787829048] req@ffff9e6803e98b40 x1874668790620160/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829048.515131:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790620160/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829048.515136:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829048.515138:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829048.515140:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790620160:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829049.025795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829049.217877:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829050.049950:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829050.241890:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829051.073787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829051.265842:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829052.097761:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829052.289912:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829053.121815:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829053.313949:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829053.633781:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829053.633791:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829053.633797:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829053.633806:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5690 00000100:00080000:0.0:1787829053.633815:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829053.633822:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829053.633853:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790620288/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829053.633874:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829053.633877:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829053.633878:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829053.633881:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5690 00000100:00080000:0.0:1787829053.633884:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829053.633887:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829053.633895:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829053.633906:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790620416/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829053.633914:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790620288:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829053.633917:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829053.633922:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829053.633924:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829053.633928:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5690 00000100:00080000:0.0:1787829053.633930:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829053.633933:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829053.633936:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829053.633946:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790620544/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829053.633958:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829053.633961:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829053.633963:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829053.633966:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5690 00000100:00080000:0.0:1787829053.633969:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829053.633972:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829053.633976:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829053.633985:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829053.633988:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790620672/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829053.633996:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829053.633998:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829053.634000:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829053.634003:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5690 00000100:00080000:0.0:1787829053.634005:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829053.634007:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829053.634010:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829053.634018:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790620800/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829053.634023:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829053/real 1787829053] req@ffff9e6803e9bc00 x1874668790620288/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829053.634028:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829053.634035:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829053.634037:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829053.634043:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5690 00000100:00080000:0.0:1787829053.634046:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829053.634049:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790620288/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829053.634053:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829053.634059:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829053.634064:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829053.634069:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790620928/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829053.634070:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829053.634078:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790620288:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829053.634085:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829053.634088:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829053.634090:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829053.634093:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5690 00000100:00080000:0.0:1787829053.634096:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829053.634098:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829053.634101:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829053.634106:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790620416:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829053.634111:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790621056/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829053.634119:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829053.634122:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829053.634123:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829053.634126:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5690 00000100:00080000:0.0:1787829053.634128:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829053.634132:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829053.634133:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829053.634139:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829053.634153:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829053/real 1787829053] req@ffff9e6803e992c0 x1874668790620416/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829053.634154:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790621184/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829053.634189:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790620416/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829053.634191:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829053.634198:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829053.634201:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829053.634209:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5690 00000100:00080000:2.0:1787829053.634210:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829053.634212:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829053.634216:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829053.634218:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829053.634223:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829053.634224:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790620416:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829053.634232:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790620544:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829053.634237:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790621312/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829053.634253:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829053.634266:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829053/real 1787829053] req@ffff9e6803e98f00 x1874668790620544/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829053.634280:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790620544/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829053.634303:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829053.634308:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829053.634311:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790620544:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829053.634320:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790620672:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829053.634348:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829053.634360:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829053/real 1787829053] req@ffff9e6803e9ad00 x1874668790620672/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829053.634370:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790620672/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829053.634379:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829053.634382:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829053.634385:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790620672:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829053.634391:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790620800:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829053.634409:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829053.634418:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829053/real 1787829053] req@ffff9e6803e9a940 x1874668790620800/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829053.634426:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790620800/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829053.634432:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829053.634433:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829053.634435:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790620800:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829053.634439:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790620928:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829053.634449:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829053.634454:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829053/real 1787829053] req@ffff9e6803e98b40 x1874668790620928/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829053.634462:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790620928/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829053.634467:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829053.634469:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829053.634472:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790620928:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829053.634484:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790621056:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829053.634495:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829053.634501:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829053/real 1787829053] req@ffff9e6803e98780 x1874668790621056/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829053.634508:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790621056/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829053.634513:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829053.634515:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829053.634516:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790621056:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829053.634520:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790621184:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829053.634530:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829053.634535:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829053/real 1787829053] req@ffff9e6803e9a1c0 x1874668790621184/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829053.634543:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790621184/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829053.634548:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829053.634549:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829053.634551:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790621184:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829053.634554:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790621312:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829053.634564:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829053.634570:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829053/real 1787829053] req@ffff9e6803e9b840 x1874668790621312/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829053.634577:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790621312/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829053.634582:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829053.634584:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829053.634585:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790621312:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829054.145814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829054.337874:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829055.169786:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829055.361879:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829056.193794:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829056.386152:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829057.217794:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829057.409924:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829058.241763:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829058.433922:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829058.753812:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829058.753820:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829058.753824:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829058.753832:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5695 00000100:00080000:3.0:1787829058.753841:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829058.753846:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829058.753886:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790621440/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829058.753905:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829058.753909:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829058.753910:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829058.753914:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5695 00000100:00080000:3.0:1787829058.753918:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829058.753923:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829058.753933:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829058.753943:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790621440:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829058.753945:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790621568/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829058.753955:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829058.753958:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829058.753960:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829058.753963:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5695 00000100:00080000:3.0:1787829058.753966:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829058.753970:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829058.753974:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829058.753982:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790621696/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829058.753990:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829058.753993:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829058.753994:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829058.753997:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5695 00000100:00080000:3.0:1787829058.754000:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829058.754001:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829058.754004:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829058.754008:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829058.754016:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790621824/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829058.754025:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829058.754028:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829058.754030:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829058.754033:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5695 00000100:00080000:3.0:1787829058.754036:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829058.754038:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829058/real 1787829058] req@ffff9e680a2d4b40 x1874668790621440/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829058.754040:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829058.754045:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829058.754054:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790621952/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829058.754063:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829058.754064:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790621440/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829058.754066:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829058.754068:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829058.754072:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5695 00000100:00080000:2.0:1787829058.754076:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829058.754076:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829058.754080:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829058.754080:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829058.754084:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829058.754085:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790621440:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829058.754094:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790622080/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829058.754102:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829058.754106:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790621568:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829058.754106:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829058.754107:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829058.754111:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5695 00000100:00080000:3.0:1787829058.754114:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829058.754117:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829058.754122:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829058.754128:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829058.754132:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790622208/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829058.754140:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829058/real 1787829058] req@ffff9e680a2d7480 x1874668790621568/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829058.754141:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829058.754145:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829058.754147:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829058.754150:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5695 00000100:00100000:2.0:1787829058.754153:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790621568/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829058.754153:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829058.754157:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829058.754162:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829058.754170:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829058.754171:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790622336/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829058.754173:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829058.754177:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790621568:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829058.754181:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829058.754183:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790621696:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829058.754184:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829058.754186:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829058.754190:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5695 00000100:00080000:3.0:1787829058.754193:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829058.754196:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787829058.754200:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787829058.754201:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829058.754211:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829058/real 1787829058] req@ffff9e680a2d5e00 x1874668790621696/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829058.754211:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790622464/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829058.754224:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790621696/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829058.754233:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829058.754236:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829058.754240:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790621696:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829058.754246:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790621824:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829058.754281:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829058.754291:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829058/real 1787829058] req@ffff9e680a2d6d00 x1874668790621824/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829058.754299:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790621824/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829058.754305:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829058.754307:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829058.754309:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790621824:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829058.754313:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790621952:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829058.754324:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829058.754330:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829058/real 1787829058] req@ffff9e680a2d61c0 x1874668790621952/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829058.754338:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790621952/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829058.754343:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829058.754344:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829058.754346:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790621952:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829058.754349:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790622080:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829058.754359:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829058.754365:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829058/real 1787829058] req@ffff9e680a2d4f00 x1874668790622080/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829058.754372:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790622080/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829058.754378:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829058.754380:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829058.754382:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790622080:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829058.754396:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790622208:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829058.754415:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829058.754422:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829058/real 1787829058] req@ffff9e680a2d7c00 x1874668790622208/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829058.754430:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790622208/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829058.754435:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829058.754437:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829058.754439:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790622208:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829058.754442:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790622336:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829058.754452:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829058.754458:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829058/real 1787829058] req@ffff9e680a2d70c0 x1874668790622336/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829058.754465:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790622336/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829058.754470:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829058.754471:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829058.754473:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790622336:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829058.754476:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790622464:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829058.754486:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829058.754491:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829058/real 1787829058] req@ffff9e680a2d6580 x1874668790622464/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829058.754497:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790622464/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829058.754503:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829058.754505:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829058.754507:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790622464:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829059.265790:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829059.457900:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829060.289808:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829060.481886:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829061.313826:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829061.505850:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829062.337805:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829062.529869:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829063.361813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829063.553935:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829063.873833:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829063.873845:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829063.873850:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829063.873862:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5700 00000100:00080000:1.0:1787829063.873875:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829063.873882:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829063.873927:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790622592/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829063.873951:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829063.873954:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829063.873956:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829063.873959:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5700 00000100:00080000:1.0:1787829063.873962:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829063.873965:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829063.873976:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829063.873998:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790622720/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829063.874005:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790622592:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829063.874010:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829063.874014:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829063.874015:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829063.874018:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5700 00000100:00080000:1.0:1787829063.874020:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829063.874022:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829063.874025:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829063.874033:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790622848/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829063.874039:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829063.874041:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829063.874043:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829063.874046:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5700 00000100:00080000:1.0:1787829063.874049:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829063.874052:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829063.874056:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829063.874063:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790622976/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829063.874069:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829063.874071:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829063.874072:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829063.874075:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5700 00000100:00080000:1.0:1787829063.874078:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829063.874082:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829063.874083:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829063.874089:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829063.874097:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790623104/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829063.874104:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829063.874107:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829063.874109:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829063.874111:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5700 00000100:00080000:1.0:1787829063.874113:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829063.874116:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829063.874119:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829063.874126:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790623232/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829063.874135:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829063/real 1787829063] req@ffff9e6831aa8b40 x1874668790622592/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829063.874153:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829063.874161:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829063.874164:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829063.874170:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5700 00000100:00080000:1.0:1787829063.874173:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829063.874175:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829063.874180:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790622592/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829063.874181:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829063.874194:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790623360/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829063.874196:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829063.874205:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829063.874207:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829063.874227:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787829063.874228:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790622592:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829063.874229:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829063.874237:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5700 00000100:00080000:1.0:1787829063.874241:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829063.874244:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829063.874247:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829063.874254:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790623488/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829063.874261:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790622720:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829063.874263:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829063.874278:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829063.874279:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829063.874285:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5700 00000100:00080000:1.0:1787829063.874288:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829063.874290:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829063.874293:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829063.874302:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790623616/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829063.874313:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829063.874329:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829063/real 1787829063] req@ffff9e6831aa9a40 x1874668790622720/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829063.874355:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790622720/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829063.874366:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829063.874369:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829063.874373:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790622720:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829063.874381:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790622848:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829063.874405:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829063.874422:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829063/real 1787829063] req@ffff9e6831aabc00 x1874668790622848/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829063.874431:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790622848/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829063.874437:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829063.874439:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829063.874442:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790622848:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829063.874446:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790622976:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829063.874461:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829063.874467:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829063/real 1787829063] req@ffff9e6831aaa940 x1874668790622976/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829063.874490:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790622976/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829063.874499:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829063.874502:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829063.874506:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790622976:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829063.874513:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790623104:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829063.874535:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829063.874555:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829063/real 1787829063] req@ffff9e6831aa8780 x1874668790623104/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829063.874569:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790623104/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829063.874577:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829063.874581:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829063.874584:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790623104:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829063.874588:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790623232:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829063.874606:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829063.874614:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829063/real 1787829063] req@ffff9e6831aaa1c0 x1874668790623232/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829063.874622:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790623232/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829063.874630:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829063.874632:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829063.874634:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790623232:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829063.874638:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790623360:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829063.874660:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829063.874665:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829063/real 1787829063] req@ffff9e6831aa8000 x1874668790623360/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829063.874673:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790623360/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829063.874683:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829063.874685:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829063.874687:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790623360:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829063.874731:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790623488:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829063.874745:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829063.874752:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829063/real 1787829063] req@ffff9e6831aab480 x1874668790623488/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829063.874760:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790623488/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829063.874766:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829063.874768:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829063.874770:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790623488:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829063.874773:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790623616:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829063.874798:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829063.874809:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829063/real 1787829063] req@ffff9e6831aaad00 x1874668790623616/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829063.874821:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790623616/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829088 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829063.874830:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829063.874834:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829063.874838:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790623616:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829064.385773:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829064.577969:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829065.409858:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829065.601865:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829066.433779:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829066.625914:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829067.457905:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829067.649908:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829068.481818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829068.673930:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829068.993803:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829068.993811:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829068.993816:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829068.993824:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5705 00000100:00080000:3.0:1787829068.993834:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829068.993840:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829068.993872:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790623744/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829068.993900:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829068.993904:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829068.993905:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829068.993910:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5705 00000100:00080000:3.0:1787829068.993914:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829068.993919:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829068.993928:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829068.993945:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790623744:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829068.993946:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790623872/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829068.993959:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829068.993962:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829068.993964:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829068.993967:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5705 00000100:00080000:3.0:1787829068.993979:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829068.993983:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829068.993987:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829068.993996:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790624000/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829068.994004:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829068.994006:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829068.994010:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829068.994011:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829068.994016:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5705 00000100:00080000:3.0:1787829068.994019:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829068.994022:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829068.994026:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829068.994034:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790624128/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829068.994046:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829068.994048:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829068.994049:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829068.994051:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829068/real 1787829068] req@ffff9e680a2d43c0 x1874668790623744/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829068.994053:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5705 00000100:00080000:3.0:1787829068.994055:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829068.994061:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829068.994068:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829068.994078:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790624256/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829068.994083:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790623744/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829068.994095:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829068.994099:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829068.994100:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829068.994105:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829068.994106:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829068.994113:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790623744:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829068.994116:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5705 00000100:00080000:3.0:1787829068.994124:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829068.994128:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829068.994133:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829068.994139:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790623872:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829068.994148:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790624384/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829068.994161:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829068.994161:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829068.994167:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829068.994170:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829068.994174:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829068/real 1787829068] req@ffff9e680a2d5a40 x1874668790623872/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829068.994191:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5705 00000100:00080000:3.0:1787829068.994200:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829068.994207:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829068.994209:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790623872/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829068.994214:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829068.994220:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829068.994224:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829068.994228:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790623872:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829068.994233:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790624512/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829068.994237:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790624000:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829068.994252:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829068.994256:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829068.994259:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829068.994264:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5705 00000800:00000100:2.0:1787829068.994269:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829068.994272:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829068.994279:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829068.994284:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829068.994291:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829068/real 1787829068] req@ffff9e680a2d4780 x1874668790624000/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829068.994301:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790624640/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829068.994311:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790624000/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829068.994312:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829068.994319:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829068.994322:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829068.994330:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5705 00000100:00080000:2.0:1787829068.994333:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829068.994334:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829068.994344:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829068.994344:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829068.994353:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829068.994355:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790624000:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829068.994363:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790624768/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829068.994368:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790624128:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829068.994402:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829068.994415:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829068/real 1787829068] req@ffff9e680a2d4000 x1874668790624128/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829068.994428:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790624128/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829068.994436:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829068.994439:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829068.994442:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790624128:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829068.994449:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790624256:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829068.994468:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829068.994480:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829068/real 1787829068] req@ffff9e680a2d52c0 x1874668790624256/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829068.994492:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790624256/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829068.994502:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829068.994505:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829068.994509:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790624256:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829068.994526:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790624384:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829068.994548:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829068.994560:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829068/real 1787829068] req@ffff9e680a2d43c0 x1874668790624384/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829068.994572:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790624384/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829068.994582:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829068.994585:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829068.994588:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790624384:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829068.994595:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790624512:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829068.994616:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829068.994628:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829068/real 1787829068] req@ffff9e680a2d6580 x1874668790624512/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829068.994640:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790624512/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829068.994648:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829068.994652:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829068.994655:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790624512:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829068.994661:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790624640:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829068.994683:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829068.994720:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829068/real 1787829068] req@ffff9e680a2d70c0 x1874668790624640/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829068.994734:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790624640/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829068.994742:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829068.994745:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829068.994749:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790624640:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829068.994755:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790624768:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829068.994778:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829068.994789:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829068/real 1787829068] req@ffff9e680a2d7c00 x1874668790624768/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829068.994802:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790624768/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829093 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829068.994810:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829068.994813:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829068.994815:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790624768:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829069.505809:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829069.697838:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829070.529800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829070.721824:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829071.553793:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829071.745850:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829072.577788:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829072.769829:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829073.601787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829073.793848:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829074.113804:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829074.113813:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829074.113817:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829074.113828:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5710 00000100:00080000:1.0:1787829074.113840:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829074.113849:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829074.113900:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790624896/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829074.113923:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829074.113927:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829074.113929:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829074.113932:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5710 00000100:00080000:1.0:1787829074.113936:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829074.113939:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829074.113948:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829074.113959:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790625024/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829074.113970:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829074.113974:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790624896:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829074.113978:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829074.113980:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829074.113985:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5710 00000100:00080000:1.0:1787829074.113990:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829074.113992:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829074.113995:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829074.114006:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790625152/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829074.114014:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829074.114017:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829074.114019:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829074.114022:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5710 00000100:00080000:1.0:1787829074.114025:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829074.114027:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829074.114031:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829074.114041:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790625280/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829074.114049:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829074.114066:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829074.114070:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829074.114071:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829074.114075:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5710 00000100:00080000:1.0:1787829074.114078:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829074.114081:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829074.114086:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829074.114095:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790625408/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829074.114103:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829074.114107:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787829074.114108:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829074/real 1787829074] req@ffff9e6831aa9680 x1874668790624896/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829074.114109:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829074.114118:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5710 00000100:00080000:1.0:1787829074.114124:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829074.114130:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829074.114135:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829074.114139:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790624896/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829074.114164:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790625536/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829074.114176:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829074.114180:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829074.114184:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829074.114185:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829074.114187:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829074.114192:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5710 00000100:00080000:1.0:1787829074.114198:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829074.114200:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790624896:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829074.114205:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829074.114211:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829074.114220:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790625664/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829074.114228:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790625024:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829074.114246:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829074.114250:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829074.114253:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829074.114259:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829074.114260:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5710 00000100:00080000:1.0:1787829074.114267:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829074.114270:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829074.114274:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829074.114280:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829074/real 1787829074] req@ffff9e6831aaa580 x1874668790625024/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829074.114287:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790625792/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829074.114300:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829074.114300:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790625024/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829074.114308:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829074.114311:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829074.114320:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5710 00000100:00080000:2.0:1787829074.114324:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829074.114325:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829074.114331:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787829074.114335:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:1.0:1787829074.114337:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829074.114346:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790625920/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829074.114354:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790625024:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829074.114366:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790625152:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829074.114394:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829074.114409:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829074/real 1787829074] req@ffff9e6831aa8f00 x1874668790625152/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829074.114430:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790625152/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829074.114443:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829074.114445:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829074.114448:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790625152:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829074.114453:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790625280:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829074.114470:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829074.114479:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829074/real 1787829074] req@ffff9e6831aa9e00 x1874668790625280/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829074.114488:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790625280/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829074.114494:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829074.114497:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829074.114499:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790625280:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829074.114503:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790625408:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829074.114514:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829074.114520:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829074/real 1787829074] req@ffff9e6831aaad00 x1874668790625408/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829074.114529:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790625408/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829074.114534:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829074.114537:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829074.114540:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790625408:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829074.114544:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790625536:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829074.114555:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829074.114561:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829074/real 1787829074] req@ffff9e6831aab480 x1874668790625536/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829074.114570:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790625536/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829074.114577:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829074.114579:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829074.114581:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790625536:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829074.114596:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790625664:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829074.114619:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829074.114627:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829074/real 1787829074] req@ffff9e6831aa8000 x1874668790625664/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829074.114636:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790625664/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829074.114642:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829074.114644:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829074.114646:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790625664:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829074.114657:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790625792:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829074.114670:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829074.114678:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829074/real 1787829074] req@ffff9e6831aaa1c0 x1874668790625792/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829074.114719:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790625792/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829074.114725:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829074.114728:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829074.114730:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790625792:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829074.114735:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790625920:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829074.114749:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829074.114757:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829074/real 1787829074] req@ffff9e6831aa8780 x1874668790625920/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829074.114765:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790625920/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829074.114771:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829074.114774:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829074.114776:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790625920:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829074.625796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829074.817874:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829075.649789:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829075.841855:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829076.673779:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829076.865903:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829077.697778:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829077.889860:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829078.721752:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829078.913851:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829079.233784:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829079.233795:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829079.233799:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829079.233809:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5715 00000100:00080000:1.0:1787829079.233819:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829079.233825:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829079.233855:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790626048/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829079.233883:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829079.233886:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829079.233887:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829079.233890:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5715 00000100:00080000:1.0:1787829079.233892:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829079.233895:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829079.233902:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829079.233913:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790626176/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829079.233921:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829079.233924:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790626048:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829079.233925:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829079.233928:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829079.233934:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5715 00000100:00080000:1.0:1787829079.233937:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829079.233939:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829079.233942:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829079.233951:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790626304/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829079.233957:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829079.233960:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829079.233961:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829079.233964:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5715 00000100:00080000:1.0:1787829079.233967:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829079.233970:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829079.233973:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829079.233981:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790626432/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829079.233986:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829079.233989:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829079.233990:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829079.233994:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5715 00000100:00080000:1.0:1787829079.233998:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829079.233998:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829079.234005:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829079.234008:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829079.234027:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790626560/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829079.234033:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829079.234036:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829079.234037:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829079.234041:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5715 00000100:00080000:1.0:1787829079.234043:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829079.234046:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829079.234050:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829079/real 1787829079] req@ffff9e6831aaa940 x1874668790626048/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829079.234051:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829079.234065:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790626688/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829079.234070:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790626048/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829079.234092:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829079.234099:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829079.234102:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829079.234103:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790626048:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829079.234121:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829079.234125:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829079.234131:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5715 00000100:00080000:1.0:1787829079.234136:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829079.234141:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790626176:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829079.234142:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829079.234152:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829079.234168:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790626816/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829079.234171:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829079.234181:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829079.234185:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829079/real 1787829079] req@ffff9e6831aabc00 x1874668790626176/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829079.234186:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829079.234190:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829079.234201:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5715 00000100:00080000:1.0:1787829079.234205:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829079.234212:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829079.234217:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790626176/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829079.234219:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829079.234229:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829079.234232:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829079.234234:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790626176:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829079.234239:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790626304:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829079.234248:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790626944/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829079.234255:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829079.234266:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829079.234266:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829079/real 1787829079] req@ffff9e6831aa9a40 x1874668790626304/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829079.234277:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829079.234282:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829079.234285:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790626304/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829079.234290:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5715 00000100:00080000:2.0:1787829079.234296:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829079.234298:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829079.234301:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829079.234306:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829079.234307:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790626304:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787829079.234311:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829079.234319:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790626432:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829079.234325:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790627072/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829079.234346:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829079.234361:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829079/real 1787829079] req@ffff9e6831aa8b40 x1874668790626432/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829079.234375:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790626432/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829079.234380:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829079.234383:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829079.234385:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790626432:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829079.234389:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790626560:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829079.234403:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829079.234410:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829079/real 1787829079] req@ffff9e6831aa8780 x1874668790626560/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829079.234417:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790626560/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829079.234422:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829079.234424:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829079.234426:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790626560:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829079.234430:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790626688:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829079.234440:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829079.234446:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829079/real 1787829079] req@ffff9e6831aaa1c0 x1874668790626688/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829079.234453:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790626688/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829079.234459:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829079.234461:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829079.234463:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790626688:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829079.234475:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790626816:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829079.234486:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829079.234491:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829079/real 1787829079] req@ffff9e6831aa8000 x1874668790626816/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829079.234499:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790626816/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829079.234504:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829079.234506:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829079.234507:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790626816:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829079.234511:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790626944:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829079.234520:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829079.234526:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829079/real 1787829079] req@ffff9e6831aab480 x1874668790626944/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829079.234533:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790626944/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829079.234542:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829079.234544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829079.234546:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790626944:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829079.234549:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790627072:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829079.234558:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829079.234563:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829079/real 1787829079] req@ffff9e6831aaad00 x1874668790627072/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829079.234570:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790627072/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829079.234575:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829079.234577:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829079.234579:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790627072:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829079.745820:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829079.937804:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829080.769810:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829080.961909:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829081.793831:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829081.985871:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829082.817789:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829083.009863:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829083.841832:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829084.033865:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829084.353812:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829084.353822:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829084.353825:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829084.353836:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5721 00000100:00080000:3.0:1787829084.353847:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829084.353855:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829084.353893:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790627200/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829084.353921:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829084.353926:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829084.353928:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829084.353933:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5721 00000100:00080000:3.0:1787829084.353937:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829084.353942:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829084.353952:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829084.353965:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790627328/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829084.353973:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829084.353975:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829084.353977:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829084.353979:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5721 00000100:00080000:3.0:1787829084.353982:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829084.353985:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829084.353988:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829084.353996:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790627456/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829084.354020:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829084.354023:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829084.354024:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829084.354029:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5721 00000100:00100000:2.0:1787829084.354032:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790627200:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829084.354032:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829084.354036:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829084.354052:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829084.354065:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790627584/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829084.354093:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829084.354097:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829084.354099:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829084.354103:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5721 00000100:00080000:3.0:1787829084.354124:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829084.354129:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787829084.354134:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787829084.354137:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829084.354156:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790627712/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829084.354167:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829084.354172:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787829084.354173:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829084/real 1787829084] req@ffff9e680a2d4f00 x1874668790627200/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829084.354176:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829084.354186:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5721 00000100:00080000:3.0:1787829084.354193:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829084.354199:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790627200/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829084.354200:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829084.354207:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829084.354219:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829084.354224:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790627840/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829084.354229:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829084.354239:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790627200:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829084.354243:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829084.354249:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829084.354251:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829084.354254:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5721 00000100:00100000:2.0:1787829084.354258:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790627328:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829084.354260:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829084.354268:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829084.354273:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829084.354288:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790627968/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829084.354298:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829084.354304:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829084.354309:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829084.354311:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829084.354318:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5721 00000100:00100000:2.0:1787829084.354319:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829084/real 1787829084] req@ffff9e680a2d61c0 x1874668790627328/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829084.354321:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829084.354328:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829084.354334:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829084.354347:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790627328/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829084.354347:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790628096/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829084.354380:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829084.354381:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829084.354387:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829084.354389:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829084.354392:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829084.354395:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5721 00000100:00080000:3.0:1787829084.354399:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829084.354404:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790627328:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829084.354404:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829084.354410:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829084.354419:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790628224/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829084.354446:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790627456:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829084.354491:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829084.354505:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829084/real 1787829084] req@ffff9e680a2d6d00 x1874668790627456/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829084.354537:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790627456/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829084.354546:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829084.354550:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829084.354553:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790627456:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829084.354561:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790627584:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829084.354582:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829084.354595:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829084/real 1787829084] req@ffff9e680a2d5e00 x1874668790627584/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829084.354607:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790627584/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829084.354616:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829084.354619:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829084.354623:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790627584:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829084.354629:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790627712:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829084.354646:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829084.354657:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829084/real 1787829084] req@ffff9e680a2d7480 x1874668790627712/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829084.354669:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790627712/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829084.354678:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829084.354681:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829084.354685:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790627712:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829084.354692:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790627840:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829084.354712:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829084.354724:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829084/real 1787829084] req@ffff9e680a2d4b40 x1874668790627840/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829084.354736:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790627840/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829084.354779:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829084.354783:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829084.354786:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790627840:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829084.354793:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790627968:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829084.354817:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829084.354830:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829084/real 1787829084] req@ffff9e680a2d7c00 x1874668790627968/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829084.354844:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790627968/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829084.354853:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829084.354856:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829084.354860:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790627968:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829084.354866:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790628096:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829084.354887:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829084.354900:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829084/real 1787829084] req@ffff9e680a2d70c0 x1874668790628096/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829084.354989:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790628096/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829084.354999:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829084.355002:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829084.355006:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790628096:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829084.355013:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790628224:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829084.355037:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829084.355051:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829084/real 1787829084] req@ffff9e680a2d6580 x1874668790628224/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829084.355063:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790628224/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829084.355085:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829084.355089:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829084.355093:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790628224:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829084.865802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829085.057867:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829085.889798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829086.081876:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829086.913794:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829087.105817:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829087.937763:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829088.129845:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829088.961812:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829089.153922:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829089.473834:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829089.473847:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829089.473853:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829089.473866:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5726 00000100:00080000:2.0:1787829089.473895:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829089.473906:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829089.473964:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790628352/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829089.474010:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829089.474017:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829089.474020:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829089.474027:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5726 00000100:00080000:2.0:1787829089.474053:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829089.474063:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829089.474076:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829089.474101:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790628480/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829089.474117:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829089.474123:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829089.474126:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829089.474133:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5726 00000100:00080000:2.0:1787829089.474141:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829089.474147:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829089.474155:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829089.474172:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790628608/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829089.474189:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829089.474194:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829089.474197:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829089.474202:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5726 00000100:00080000:2.0:1787829089.474206:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829089.474212:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829089.474218:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829089.474235:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790628736/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829089.474251:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829089.474257:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829089.474260:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829089.474267:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790628352:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829089.474271:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5726 00000100:00080000:2.0:1787829089.474281:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829089.474288:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829089.474298:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829089.474322:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790628864/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829089.474340:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829089.474346:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829089.474350:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829089.474357:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5726 00000100:00080000:2.0:1787829089.474366:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829089.474373:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829089.474382:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829089.474400:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790628992/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829089.474414:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787829089.474421:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829089.474424:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829089.474428:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829089.474436:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5726 00000100:00080000:2.0:1787829089.474442:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829089.474447:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829089.474455:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829089.474469:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829089/real 1787829089] req@ffff9e68044212c0 x1874668790628352/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829089.474473:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790629120/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829089.474500:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829089.474502:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829089.474504:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829089.474507:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5726 00000100:00080000:2.0:1787829089.474509:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829089.474512:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829089.474515:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829089.474522:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790629248/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829089.474530:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829089.474533:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829089.474534:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829089.474537:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5726 00000100:00080000:2.0:1787829089.474540:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829089.474542:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829089.474545:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829089.474552:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790629376/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829089.474720:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790628352/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829089.474739:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829089.474744:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829089.474750:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790628352:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829089.474824:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790628480:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829089.474864:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829089.474884:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829089/real 1787829089] req@ffff9e68044230c0 x1874668790628480/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829089.474918:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790628480/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829089.474927:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829089.474930:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829089.474934:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790628480:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829089.474940:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790628608:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829089.474961:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829089.474988:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829089/real 1787829089] req@ffff9e6804420b40 x1874668790628608/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829089.474998:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790628608/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829089.475005:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829089.475008:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829089.475011:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790628608:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829089.475016:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790628736:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829089.475032:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829089.475062:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829089/real 1787829089] req@ffff9e6804421e00 x1874668790628736/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829089.475090:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790628736/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829089.475098:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829089.475101:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829089.475104:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790628736:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829089.475130:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790628864:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829089.475150:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829089.475160:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829089/real 1787829089] req@ffff9e6804423840 x1874668790628864/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829089.475176:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790628864/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829089.475189:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829089.475193:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829089.475198:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790628864:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829089.475207:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790628992:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829089.475235:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829089.475252:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829089/real 1787829089] req@ffff9e68044221c0 x1874668790628992/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829089.475286:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790628992/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829089.475299:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829089.475303:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829089.475308:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790628992:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829089.475315:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790629120:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829089.475336:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829089.475347:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829089/real 1787829089] req@ffff9e6804420780 x1874668790629120/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829089.475356:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790629120/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829089.475379:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829089.475382:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829089.475385:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790629120:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829089.475389:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790629248:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829089.475403:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829089.475410:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829089/real 1787829089] req@ffff9e6804422d00 x1874668790629248/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829089.475419:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790629248/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829089.475430:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829089.475432:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829089.475435:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790629248:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829089.475439:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790629376:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829089.475451:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829089.475463:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829089/real 1787829089] req@ffff9e6804422940 x1874668790629376/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829089.475477:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790629376/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829089.475486:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829089.475490:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829089.475493:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790629376:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829089.985764:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829090.177811:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829091.009785:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829091.201889:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829092.033773:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829092.225836:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829093.057795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829093.249886:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829094.081763:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829094.273817:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829094.593786:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829094.593797:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829094.593801:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829094.593811:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5731 00000100:00080000:1.0:1787829094.593822:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829094.593829:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829094.593875:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790629504/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829094.593897:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829094.593900:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829094.593901:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829094.593904:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5731 00000100:00080000:1.0:1787829094.593907:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829094.593911:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829094.593918:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829094.593927:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790629632/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829094.593939:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829094.593943:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829094.593944:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829094.593946:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5731 00000100:00080000:1.0:1787829094.593949:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829094.593951:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829094.593953:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829094.593959:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790629760/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829094.593966:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829094.593969:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829094.593970:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829094.593973:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5731 00000100:00080000:1.0:1787829094.593974:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829094.593977:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829094.593980:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829094.593991:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790629888/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829094.593991:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790629504:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829094.594020:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829094.594022:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829094.594024:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829094.594026:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5731 00000100:00080000:1.0:1787829094.594028:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829094.594031:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829094.594035:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829094.594055:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790630016/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829094.594061:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829094.594064:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829094.594065:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829094.594067:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5731 00000100:00080000:1.0:1787829094.594069:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829094.594072:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829094.594075:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829094.594082:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790630144/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829094.594088:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829094.594095:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829094.594098:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829094.594100:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829094.594103:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5731 00000100:00080000:1.0:1787829094.594109:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829094.594111:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829094.594114:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829094.594120:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790630272/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829094.594127:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829094.594129:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829094/real 1787829094] req@ffff9e6831aa9e00 x1874668790629504/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829094.594131:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829094.594134:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829094.594142:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5731 00000100:00080000:1.0:1787829094.594147:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829094.594150:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829094.594154:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829094.594156:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790629504/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829094.594165:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790630400/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829094.594168:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829094.594177:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829094.594178:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829094.594182:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829094.594185:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829094.594186:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790629504:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829094.594192:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5731 00000100:00080000:1.0:1787829094.594195:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829094.594198:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790629632:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829094.594206:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829094.594212:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787829094.594221:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829094.594224:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790630528/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829094.594235:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829094/real 1787829094] req@ffff9e6831aa8f00 x1874668790629632/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829094.594246:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790629632/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829094.594253:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829094.594256:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829094.594258:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790629632:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829094.594262:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790629760:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829094.594272:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829094.594279:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829094/real 1787829094] req@ffff9e6831aaa580 x1874668790629760/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829094.594290:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790629760/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829094.594295:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829094.594297:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829094.594298:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790629760:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829094.594318:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790629888:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829094.594328:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829094.594334:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829094/real 1787829094] req@ffff9e6831aa9680 x1874668790629888/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829094.594342:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790629888/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829094.594350:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829094.594354:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829094.594357:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790629888:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829094.594363:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790630016:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829094.594383:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829094.594394:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829094/real 1787829094] req@ffff9e6831aaad00 x1874668790630016/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829094.594407:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790630016/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829094.594419:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829094.594421:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829094.594423:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790630016:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829094.594427:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790630144:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829094.594442:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829094.594449:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829094/real 1787829094] req@ffff9e6831aab480 x1874668790630144/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829094.594456:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790630144/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829094.594463:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829094.594464:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829094.594466:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790630144:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829094.594469:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790630272:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829094.594479:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829094.594485:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829094/real 1787829094] req@ffff9e6831aa8000 x1874668790630272/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829094.594492:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790630272/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829094.594497:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829094.594499:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829094.594501:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790630272:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829094.594504:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790630400:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829094.594514:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829094.594519:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829094/real 1787829094] req@ffff9e6831aaa1c0 x1874668790630400/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829094.594525:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790630400/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829094.594539:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829094.594541:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829094.594543:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790630400:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829094.594546:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790630528:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829094.594555:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829094.594561:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829094/real 1787829094] req@ffff9e6831aa8780 x1874668790630528/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829094.594567:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790630528/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829094.594571:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829094.594573:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829094.594575:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790630528:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829095.105811:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829095.297936:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829096.129787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829096.321900:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829097.153803:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829097.345868:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829098.177753:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829098.369886:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829099.201798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829099.393868:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829099.714751:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829099.714760:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829099.714763:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829099.714772:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5736 00000100:00080000:1.0:1787829099.714781:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829099.714787:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829099.714818:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790630656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829099.714844:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829099.714847:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829099.714848:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829099.714851:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5736 00000100:00080000:1.0:1787829099.714853:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829099.714856:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829099.714863:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829099.714873:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790630784/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829099.714881:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829099.714885:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1787829099.714885:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790630656:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829099.714888:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829099.714894:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5736 00000100:00080000:1.0:1787829099.714897:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829099.714899:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829099.714902:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829099.714912:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790630912/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829099.714918:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829099.714920:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829099.714922:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829099.714925:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5736 00000100:00080000:1.0:1787829099.714928:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829099.714931:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829099.714935:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829099.714943:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790631040/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829099.714949:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829099.714952:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829099.714953:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829099.714955:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5736 00000100:00080000:1.0:1787829099.714957:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829099.714959:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829099.714962:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829099.714987:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790631168/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829099.714995:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790630784:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829099.714996:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829099.715002:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829099.715005:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829099.715009:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5736 00000100:00080000:1.0:1787829099.715012:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829099.715015:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829099.715020:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829099.715029:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790631296/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829099.715038:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790630912:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829099.715045:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829099.715048:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829099.715050:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829099.715053:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5736 00000100:00080000:1.0:1787829099.715057:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829099.715060:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829099.715064:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829099.715069:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790631040:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829099.715074:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790631424/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829099.715082:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829099.715084:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829099.715086:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829099.715089:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5736 00000100:00080000:1.0:1787829099.715093:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829099.715096:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829099.715099:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790631168:192.168.203.109@tcp:8:kworker.0 00000008:00100000:1.0:1787829099.715101:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829099.715113:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790631552/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829099.715121:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829099.715125:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790631296:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829099.715126:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829099.715129:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829099.715137:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5736 00000100:00080000:1.0:1787829099.715144:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829099.715149:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829099.715152:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790631424:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787829099.715158:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829099.715175:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790631552:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829099.715177:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790631680/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829099.715196:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790631680:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829100.225758:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829100.417864:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829101.249805:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829101.441844:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829102.273799:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829102.465841:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787829102.786037:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:1.0:1787829102.786079:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787829102.786097:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829102.786147:0:2387:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:1.0:1787829102.786160:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829102.786170:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829102.786178:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829102.786186:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829102.786194:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829102.786202:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829102.786209:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:3.0:1787829102.786287:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829099/real 1787829102] req@ffff9e6831aaad00 x1874668790631680/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829102.786326:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790631680/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829102.786339:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829102.786345:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829102.786349:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790631680:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829102.786357:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829099/real 1787829102] req@ffff9e6831aab480 x1874668790631552/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829102.786371:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790631552/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829102.786380:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829102.786384:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829102.786388:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790631552:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829102.786395:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829099/real 1787829102] req@ffff9e6831aa8780 x1874668790631168/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829102.786408:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790631168/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829102.786416:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829102.786419:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829102.786423:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790631168:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829102.786432:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829099/real 1787829102] req@ffff9e6831aaa1c0 x1874668790631296/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829102.786443:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790631296/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829102.786453:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829102.786456:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829102.786459:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790631296:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829102.786466:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829099/real 1787829102] req@ffff9e6831aa8000 x1874668790631424/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829102.786478:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790631424/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829102.786486:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829102.786490:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829102.786493:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790631424:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829102.786501:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829099/real 1787829102] req@ffff9e6831aa9a40 x1874668790630784/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829102.786513:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790630784/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829102.786522:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829102.786526:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829102.786530:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790630784:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829102.786537:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829099/real 1787829102] req@ffff9e6831aabc00 x1874668790630912/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829102.786549:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790630912/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829102.786558:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829102.786562:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829102.786565:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790630912:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829102.786572:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829099/real 1787829102] req@ffff9e6831aaa940 x1874668790631040/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829102.786584:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790631040/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829102.786592:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829102.786595:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829102.786599:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790631040:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829102.786607:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829099/real 1787829102] req@ffff9e6831aa8b40 x1874668790630656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829102.786619:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790630656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829102.786626:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829102.786629:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829102.786632:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790630656:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829103.297773:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829103.489856:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829104.321810:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829104.513896:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829104.833861:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829104.833873:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829104.833880:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829104.833892:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5741 00000100:00080000:2.0:1787829104.833903:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829104.833912:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829104.833985:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790631808/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829104.834014:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829104.834019:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829104.834021:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829104.834026:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5741 00000100:00080000:2.0:1787829104.834030:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829104.834037:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829104.834047:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829104.834062:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790631808:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829104.834064:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790631936/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829104.834081:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829104.834085:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829104.834087:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829104.834092:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5741 00000100:00080000:2.0:1787829104.834095:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829104.834100:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829104.834106:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829104.834118:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790632064/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829104.834136:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787829104.834136:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829104.834143:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829104.834145:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829104.834151:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5741 00000100:00080000:2.0:1787829104.834156:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829104.834161:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829104.834167:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829104.834180:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829104/real 1787829104] req@ffff9e6804420f00 x1874668790631808/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829104.834183:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790632192/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829104.834215:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829104.834220:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829104.834222:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829104.834225:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790631808/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829104.834228:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5741 00000100:00080000:2.0:1787829104.834242:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829104.834243:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829104.834249:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:3.0:1787829104.834252:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:2.0:1787829104.834258:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829104.834259:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790631808:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829104.834275:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790632320/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829104.834286:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829104.834290:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1787829104.834293:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790631936:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829104.834294:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829104.834303:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5741 00000100:00080000:2.0:1787829104.834309:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829104.834314:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829104.834319:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829104.834330:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829104.834335:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790632448/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829104.834347:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829104/real 1787829104] req@ffff9e6804422580 x1874668790631936/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829104.834349:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829104.834358:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829104.834362:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829104.834375:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790631936/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829104.834379:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5741 00000100:00080000:2.0:1787829104.834388:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829104.834394:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829104.834395:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:3.0:1787829104.834405:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:2.0:1787829104.834406:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829104.834413:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790631936:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829104.834424:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790632576/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829104.834442:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829104.834442:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790632064:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829104.834458:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829104.834461:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829104.834467:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5741 00000100:00080000:2.0:1787829104.834472:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829104.834477:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829104.834481:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829104.834489:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829104.834494:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829104/real 1787829104] req@ffff9e6804421680 x1874668790632064/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829104.834506:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790632704/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829104.834511:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790632064/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829104.834538:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829104.834545:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829104.834546:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829104.834553:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790632064:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829104.834554:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829104.834558:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829104.834565:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5741 00000100:00100000:3.0:1787829104.834567:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790632192:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787829104.834571:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829104.834577:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829104.834583:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787829104.834595:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829104.834599:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790632832/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829104.834613:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829104/real 1787829104] req@ffff9e6804423480 x1874668790632192/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829104.834627:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790632192/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829104.834636:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829104.834640:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829104.834643:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790632192:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829104.834649:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790632320:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829104.834669:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829104.834681:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829104/real 1787829104] req@ffff9e6804422940 x1874668790632320/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829104.834723:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790632320/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829104.834733:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829104.834737:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829104.834740:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790632320:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829104.834760:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790632448:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829104.834783:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829104.834796:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829104/real 1787829104] req@ffff9e6804422d00 x1874668790632448/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829104.834808:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790632448/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829104.834818:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829104.834821:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829104.834823:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790632448:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829104.834830:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790632576:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829104.834849:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829104.835110:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829104/real 1787829104] req@ffff9e6804420780 x1874668790632576/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829104.835127:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790632576/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829104.835143:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829104.835153:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829104.835157:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790632576:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829104.835164:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790632704:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829104.835188:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829104.835201:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829104/real 1787829104] req@ffff9e68044221c0 x1874668790632704/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829104.835213:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790632704/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829104.835221:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829104.835224:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829104.835228:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790632704:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829104.835233:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790632832:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829104.835257:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829104.835268:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829104/real 1787829104] req@ffff9e6804423840 x1874668790632832/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829104.835278:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790632832/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829104.835285:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829104.835288:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829104.835292:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790632832:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829105.345776:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829105.537852:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829106.369781:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829106.561890:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829107.393783:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829107.585859:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829108.417832:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829108.609872:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829109.441769:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829109.633857:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829109.953766:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829109.953774:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829109.953778:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829109.953787:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5746 00000100:00080000:2.0:1787829109.953797:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829109.953803:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829109.953835:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790632960/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829109.953856:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829109.953859:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829109.953860:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829109.953863:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5746 00000100:00080000:2.0:1787829109.953865:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829109.953868:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829109.953876:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829109.953884:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790633088/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829109.953904:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829109.953907:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829109.953910:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829109.953914:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790632960:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829109.953916:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5746 00000100:00080000:2.0:1787829109.953922:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829109.953926:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829109.953930:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829109.953940:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790633216/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829109.953948:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829109.953952:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829109.953954:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829109.953958:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5746 00000100:00080000:2.0:1787829109.953964:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829109.953969:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829109.953974:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787829109.953976:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829109.953993:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790633344/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829109.954003:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829109.954007:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829109.954010:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829109.954014:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829109/real 1787829109] req@ffff9e6804421e00 x1874668790632960/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829109.954018:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5746 00000100:00080000:2.0:1787829109.954026:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829109.954032:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829109.954036:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790632960/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829109.954040:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787829109.954050:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829109.954055:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829109.954060:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790633472/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829109.954061:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790632960:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829109.954076:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829109.954079:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829109.954080:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829109.954084:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5746 00000100:00080000:2.0:1787829109.954089:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829109.954093:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829109.954096:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790633088:192.168.203.109@tcp:38:kworker.0 00000008:00100000:2.0:1787829109.954098:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829109.954118:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790633600/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829109.954123:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829109.954133:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829109.954135:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829109.954139:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829109.954139:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829109/real 1787829109] req@ffff9e6804420b40 x1874668790633088/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829109.954147:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5746 00000100:00080000:2.0:1787829109.954153:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829109.954158:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829109.954162:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829109.954162:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790633088/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829109.954174:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790633728/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829109.954174:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829109.954181:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829109.954189:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790633088:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829109.954190:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829109.954197:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829109.954198:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829109.954201:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790633216:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829109.954204:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5746 00000100:00080000:2.0:1787829109.954210:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829109.954212:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829109.954216:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787829109.954224:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829109.954227:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790633856/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829109.954236:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829109.954236:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829109/real 1787829109] req@ffff9e68044230c0 x1874668790633216/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829109.954243:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829109.954246:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829109.954254:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5746 00000100:00100000:3.0:1787829109.954257:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790633216/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829109.954258:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829109.954264:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829109.954270:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787829109.954272:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829109.954277:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829109.954280:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790633984/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829109.954283:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790633216:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829109.954297:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790633344:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829109.954312:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829109.954319:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829109/real 1787829109] req@ffff9e68044212c0 x1874668790633344/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829109.954326:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790633344/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829109.954332:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829109.954334:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829109.954336:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790633344:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829109.954340:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790633472:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829109.954350:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829109.954355:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829109/real 1787829109] req@ffff9e6804423840 x1874668790633472/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829109.954362:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790633472/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829109.954367:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829109.954369:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829109.954371:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790633472:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829109.954381:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790633600:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829109.954391:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829109.954397:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829109/real 1787829109] req@ffff9e68044221c0 x1874668790633600/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829109.954405:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790633600/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829109.954411:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829109.954412:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829109.954414:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790633600:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829109.954418:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790633728:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829109.954427:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829109.954433:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829109/real 1787829109] req@ffff9e6804420780 x1874668790633728/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829109.954444:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790633728/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829109.954452:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829109.954455:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829109.954458:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790633728:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829109.954464:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790633856:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829109.954485:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829109.954497:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829109/real 1787829109] req@ffff9e6804422d00 x1874668790633856/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829109.954506:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790633856/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829109.954512:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829109.954514:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829109.954516:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790633856:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829109.954526:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790633984:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829109.954541:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829109.954548:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829109/real 1787829109] req@ffff9e6804422940 x1874668790633984/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829109.954555:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790633984/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829134 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829109.954561:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829109.954563:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829109.954565:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790633984:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829110.465789:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829110.657886:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829111.489806:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829111.681830:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829112.513837:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829112.705923:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829113.537781:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829113.729823:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829114.561784:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829114.753928:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829115.073803:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829115.073812:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829115.073815:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829115.073825:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5751 00000100:00080000:2.0:1787829115.073834:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829115.073840:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829115.073889:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790634112/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829115.073916:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829115.073919:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829115.073922:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829115.073925:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5751 00000100:00080000:2.0:1787829115.073928:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829115.073931:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829115.073938:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829115.073951:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790634240/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829115.073959:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829115.073961:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829115.073963:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829115.073965:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5751 00000100:00080000:2.0:1787829115.073968:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829115.073970:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829115.073973:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829115.073980:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790634112:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829115.073982:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790634368/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829115.073999:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829115.074002:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829115.074004:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829115.074008:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5751 00000100:00080000:2.0:1787829115.074012:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829115.074016:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829115.074021:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829115.074035:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790634496/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829115.074041:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829115.074052:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829115.074056:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829115.074057:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829115.074062:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5751 00000100:00080000:2.0:1787829115.074067:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829115.074072:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829115.074077:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829115.074080:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829115/real 1787829115] req@ffff9e6804423480 x1874668790634112/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829115.074095:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790634624/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829115.074099:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790634112/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829115.074124:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829115.074125:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829115.074128:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829115.074132:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829115.074133:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829115.074140:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5751 00000100:00100000:3.0:1787829115.074142:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790634112:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829115.074146:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829115.074151:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790634240:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829115.074152:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829115.074161:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829115.074173:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790634752/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829115.074178:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829115.074187:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829115.074189:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829115.074191:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829115.074194:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829115/real 1787829115] req@ffff9e6804421680 x1874668790634240/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829115.074195:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5751 00000100:00080000:2.0:1787829115.074200:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829115.074205:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829115.074212:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829115.074217:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790634240/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829115.074224:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790634880/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829115.074228:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829115.074235:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829115.074236:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829115.074242:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787829115.074246:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790634240:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829115.074247:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829115.074254:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5751 00000100:00080000:2.0:1787829115.074256:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829115.074259:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829115.074262:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829115.074268:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790635008/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829115.074273:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790634368:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829115.074288:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829115.074291:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829115.074292:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829115.074298:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5751 00000800:00000100:3.0:1787829115.074299:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829115.074302:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829115.074307:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829115.074310:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829115.074313:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829115/real 1787829115] req@ffff9e6804422580 x1874668790634368/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829115.074321:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790635136/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829115.074330:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790634368/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829115.074338:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829115.074340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829115.074343:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790634368:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829115.074347:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790634496:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829115.074360:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829115.074366:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829115/real 1787829115] req@ffff9e6804420f00 x1874668790634496/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829115.074373:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790634496/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829115.074378:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829115.074380:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829115.074383:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790634496:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829115.074386:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790634624:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829115.074395:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829115.074401:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829115/real 1787829115] req@ffff9e6804422940 x1874668790634624/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829115.074407:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790634624/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829115.074413:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829115.074414:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829115.074416:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790634624:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829115.074419:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790634752:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829115.074429:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829115.074434:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829115/real 1787829115] req@ffff9e6804422d00 x1874668790634752/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829115.074441:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790634752/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829115.074447:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829115.074449:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829115.074451:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790634752:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829115.074454:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790634880:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829115.074463:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829115.074469:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829115/real 1787829115] req@ffff9e6804420780 x1874668790634880/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829115.074475:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790634880/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829115.074480:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829115.074481:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829115.074483:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790634880:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829115.074496:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790635008:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829115.074505:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829115.074511:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829115/real 1787829115] req@ffff9e68044221c0 x1874668790635008/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829115.074518:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790635008/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829115.074523:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829115.074524:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829115.074526:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790635008:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829115.074529:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790635136:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829115.074539:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829115.074545:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829115/real 1787829115] req@ffff9e6804423840 x1874668790635136/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829115.074552:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790635136/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829115.074556:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829115.074558:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829115.074560:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790635136:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829115.585788:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829115.777935:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829116.609761:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829116.801880:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829117.633789:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829117.825889:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829118.657799:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829118.849847:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829119.681862:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829119.873860:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829120.193809:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829120.193819:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829120.193834:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829120.193845:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5756 00000100:00080000:1.0:1787829120.193856:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829120.193864:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829120.193899:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790635264/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829120.193923:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829120.193926:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829120.193928:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829120.193931:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5756 00000100:00080000:1.0:1787829120.193933:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829120.193937:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829120.193945:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829120.193957:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790635392/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829120.193960:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790635264:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829120.193976:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829120.193981:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829120.193983:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829120.193988:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5756 00000100:00080000:1.0:1787829120.193992:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829120.193997:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829120.194003:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829120.194013:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790635520/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829120.194021:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829120.194023:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829120.194025:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829120.194030:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5756 00000800:00000100:3.0:1787829120.194031:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829120.194034:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829120.194039:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829120.194044:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829120.194053:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790635648/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829120.194059:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829120.194062:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829120.194063:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829120.194066:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5756 00000100:00080000:1.0:1787829120.194068:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829120.194071:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829120.194074:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829120.194079:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829120/real 1787829120] req@ffff9e6831aa9680 x1874668790635264/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829120.194084:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790635776/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829120.194110:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829120.194113:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829120.194117:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829120.194117:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790635264/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829120.194127:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5756 00000100:00080000:3.0:1787829120.194134:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829120.194135:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829120.194143:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829120.194146:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829120.194150:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790635264:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787829120.194154:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829120.194168:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790635904/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829120.194174:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790635392:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829120.194189:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829120.194192:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829120.194194:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829120.194198:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5756 00000100:00080000:1.0:1787829120.194201:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829120.194206:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829120.194208:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829120.194213:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829120.194221:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829120/real 1787829120] req@ffff9e6831aaa580 x1874668790635392/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829120.194225:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790636032/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829120.194250:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829120.194254:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787829120.194254:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790635392/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829120.194258:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829120.194266:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5756 00000100:00080000:1.0:1787829120.194271:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829120.194273:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829120.194276:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:3.0:1787829120.194280:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:1.0:1787829120.194285:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829120.194285:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790635392:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829120.194293:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790635520:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829120.194302:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790636160/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829120.194310:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829120.194319:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829120.194324:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829120/real 1787829120] req@ffff9e6831aa8f00 x1874668790635520/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829120.194327:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829120.194332:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829120.194340:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5756 00000100:00100000:3.0:1787829120.194342:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790635520/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829120.194345:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829120.194353:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:3.0:1787829120.194355:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787829120.194358:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787829120.194360:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829120.194365:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790635520:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829120.194373:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790636288/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829120.194374:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790635648:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829120.194396:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829120.194405:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829120/real 1787829120] req@ffff9e6831aa9e00 x1874668790635648/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829120.194413:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790635648/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829120.194418:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829120.194420:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829120.194422:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790635648:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829120.194426:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790635776:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829120.194436:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829120.194441:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829120/real 1787829120] req@ffff9e6831aa8b40 x1874668790635776/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829120.194448:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790635776/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829120.194454:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829120.194455:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829120.194457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790635776:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829120.194468:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790635904:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829120.194478:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829120.194484:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829120/real 1787829120] req@ffff9e6831aaa940 x1874668790635904/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829120.194500:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790635904/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829120.194515:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829120.194517:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829120.194519:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790635904:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829120.194523:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790636032:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829120.194534:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829120.194540:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829120/real 1787829120] req@ffff9e6831aabc00 x1874668790636032/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829120.194547:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790636032/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829120.194552:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829120.194553:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829120.194555:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790636032:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829120.194559:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790636160:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829120.194575:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829120.194586:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829120/real 1787829120] req@ffff9e6831aa9a40 x1874668790636160/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829120.194598:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790636160/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829120.194606:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829120.194610:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829120.194614:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790636160:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829120.194621:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790636288:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829120.194637:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829120.194645:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829120/real 1787829120] req@ffff9e6831aa8000 x1874668790636288/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829120.194654:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790636288/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829120.194663:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829120.194666:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829120.194669:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790636288:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829120.705801:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829120.897864:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829121.729799:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829121.921854:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829122.753794:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829122.945951:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829123.777792:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829123.969847:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829124.801830:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829124.993907:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829125.313855:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829125.313863:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829125.313866:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829125.313875:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5762 00000100:00080000:2.0:1787829125.313902:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829125.313909:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829125.313943:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790636416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829125.313976:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829125.313981:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829125.313983:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829125.313987:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5762 00000100:00080000:2.0:1787829125.313990:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829125.313993:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829125.314003:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829125.314021:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790636544/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829125.314034:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829125.314035:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790636416:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829125.314041:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829125.314046:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829125.314051:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5762 00000100:00080000:2.0:1787829125.314055:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829125.314060:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829125.314065:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829125.314098:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790636672/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829125.314109:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829125.314113:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829125.314116:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829125.314120:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5762 00000100:00080000:2.0:1787829125.314125:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829125.314128:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829125.314134:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829125.314141:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829125.314155:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790636800/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829125.314164:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829125.314165:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829125/real 1787829125] req@ffff9e68044212c0 x1874668790636416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829125.314174:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829125.314178:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829125.314187:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5762 00000100:00080000:2.0:1787829125.314191:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829125.314196:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829125.314197:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790636416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829125.314204:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829125.314215:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829125.314221:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829125.314221:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790636928/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829125.314230:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790636416:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829125.314237:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829125.314241:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829125.314242:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829125.314246:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5762 00000100:00080000:2.0:1787829125.314249:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829125.314252:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829125.314255:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829125.314257:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790636544:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829125.314266:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790637056/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829125.314275:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829125.314277:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829125.314279:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829125.314284:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829125.314287:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5762 00000100:00080000:2.0:1787829125.314292:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829125.314295:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829125.314299:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829125.314301:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829125/real 1787829125] req@ffff9e68044230c0 x1874668790636544/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829125.314309:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790637184/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829125.314318:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829125.314320:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790636544/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829125.314324:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829125.314327:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829125.314335:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5762 00000100:00080000:0.0:1787829125.314338:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829125.314340:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829125.314348:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829125.314348:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829125.314358:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829125.314359:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790636544:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829125.314375:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790636672:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829125.314375:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790637312/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829125.314392:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829125.314395:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829125.314399:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829125.314403:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5762 00000100:00080000:2.0:1787829125.314406:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829125.314408:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829125.314412:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829125.314417:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829125.314430:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829125/real 1787829125] req@ffff9e6804420b40 x1874668790636672/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829125.314433:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790637440/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829125.314481:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790636672/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829125.314490:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829125.314493:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829125.314498:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790636672:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829125.314505:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790636800:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829125.314523:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829125.314534:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829125/real 1787829125] req@ffff9e6804421e00 x1874668790636800/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829125.314547:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790636800/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829125.314564:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829125.314569:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829125.314573:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790636800:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829125.314581:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790636928:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829125.314601:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829125.314610:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829125/real 1787829125] req@ffff9e6804423840 x1874668790636928/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829125.314618:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790636928/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829125.314623:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829125.314625:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829125.314628:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790636928:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829125.314640:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790637056:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829125.314650:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829125.314656:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829125/real 1787829125] req@ffff9e68044221c0 x1874668790637056/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829125.314664:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790637056/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829125.314670:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829125.314671:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829125.314674:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790637056:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829125.314678:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790637184:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829125.314687:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829125.314693:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829125/real 1787829125] req@ffff9e6804420780 x1874668790637184/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829125.314729:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790637184/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829125.314735:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829125.314738:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829125.314742:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790637184:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829125.314750:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790637312:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829125.314763:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829125.314770:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829125/real 1787829125] req@ffff9e6804422d00 x1874668790637312/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829125.314778:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790637312/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829125.314782:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829125.314784:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829125.314802:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790637312:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829125.314807:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790637440:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829125.314840:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829125.314850:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829125/real 1787829125] req@ffff9e6804422940 x1874668790637440/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829125.314860:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790637440/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829125.314866:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829125.314868:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829125.314870:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790637440:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829125.825808:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829126.017898:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829126.849791:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829127.041885:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829127.873814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829128.065845:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829128.897827:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829129.089837:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829129.921813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829130.113865:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829130.433790:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829130.433798:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829130.433802:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829130.433811:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5767 00000100:00080000:2.0:1787829130.433820:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829130.433825:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829130.433862:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790637568/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829130.433880:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829130.433883:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829130.433885:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829130.433890:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5767 00000100:00080000:2.0:1787829130.433894:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829130.433899:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829130.433908:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829130.433924:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790637568:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829130.433925:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790637696/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829130.433941:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829130.433945:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829130.433955:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829130.433959:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5767 00000100:00080000:2.0:1787829130.433963:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829130.433966:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829130.433971:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829130.433981:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790637824/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829130.433992:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829130.433999:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829130.434002:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829130.434003:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829130.434007:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5767 00000100:00080000:2.0:1787829130.434010:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829130.434013:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829130.434016:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829130.434026:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790637952/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829130.434028:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829130/real 1787829130] req@ffff9e6804420f00 x1874668790637568/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829130.434046:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829130.434049:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829130.434051:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829130.434056:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790637568/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829130.434056:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5767 00000100:00080000:2.0:1787829130.434058:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829130.434065:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829130.434072:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829130.434075:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829130.434080:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829130.434083:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790638080/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829130.434086:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790637568:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829130.434101:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829130.434105:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829130.434108:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829130.434111:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790637696:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829130.434117:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5767 00000100:00080000:2.0:1787829130.434123:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829130.434128:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787829130.434133:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:2.0:1787829130.434137:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829130.434144:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829130/real 1787829130] req@ffff9e6804422580 x1874668790637696/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829130.434154:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790638208/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829130.434156:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790637696/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829130.434190:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829130.434191:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829130.434196:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829130.434197:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829130.434201:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829130.434202:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790637696:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829130.434207:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5767 00000100:00100000:0.0:1787829130.434211:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790637824:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829130.434213:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829130.434220:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829130.434224:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829130.434230:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829130.434238:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790638336/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829130.434242:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829130/real 1787829130] req@ffff9e6804421680 x1874668790637824/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829130.434258:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829130.434262:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790637824/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829130.434264:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829130.434267:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829130.434276:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5767 00000100:00080000:0.0:1787829130.434278:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829130.434280:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829130.434283:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829130.434287:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829130.434290:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790637824:192.168.203.109@tcp:38:kworker.0 00000008:00100000:2.0:1787829130.434293:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829130.434297:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790637952:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829130.434306:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790638464/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829130.434313:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829130.434322:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829130/real 1787829130] req@ffff9e6804423480 x1874668790637952/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829130.434349:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829130.434350:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790637952/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829130.434357:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829130.434361:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829130.434368:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5767 00000100:00080000:0.0:1787829130.434370:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829130.434372:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829130.434378:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829130.434379:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829130.434385:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829130.434388:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790637952:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829130.434397:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790638592/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829130.434400:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790638080:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829130.434430:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829130.434439:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829130/real 1787829130] req@ffff9e6804422940 x1874668790638080/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829130.434447:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790638080/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829130.434453:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829130.434455:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829130.434457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790638080:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829130.434469:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790638208:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829130.434480:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829130.434486:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829130/real 1787829130] req@ffff9e6804422d00 x1874668790638208/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829130.434493:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790638208/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829130.434499:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829130.434501:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829130.434503:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790638208:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829130.434506:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790638336:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829130.434516:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829130.434522:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829130/real 1787829130] req@ffff9e6804420780 x1874668790638336/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829130.434529:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790638336/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829130.434534:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829130.434536:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829130.434538:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790638336:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829130.434541:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790638464:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829130.434550:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829130.434555:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829130/real 1787829130] req@ffff9e68044221c0 x1874668790638464/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829130.434561:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790638464/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829130.434566:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829130.434568:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829130.434570:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790638464:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829130.434573:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790638592:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829130.434582:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829130.434587:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829130/real 1787829130] req@ffff9e6804423840 x1874668790638592/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829130.434594:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790638592/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829130.434600:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829130.434602:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829130.434604:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790638592:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829130.945889:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829131.137871:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829131.969805:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829132.161913:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829132.993803:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829133.186328:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829134.017835:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829134.209869:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829135.041800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829135.233880:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829135.553781:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829135.553801:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829135.553805:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829135.553813:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5772 00000100:00080000:2.0:1787829135.553822:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829135.553828:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829135.553860:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790638720/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829135.553883:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829135.553887:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829135.553889:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829135.553893:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5772 00000100:00080000:2.0:1787829135.553898:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829135.553903:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829135.553914:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829135.553922:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790638720:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829135.553931:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790638848/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829135.553944:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829135.553948:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829135.553950:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829135.553954:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5772 00000100:00080000:2.0:1787829135.553959:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829135.553974:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829135.553979:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829135.553992:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829135.553994:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790638976/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829135.554004:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829135.554007:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829135.554008:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829135.554011:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5772 00000100:00080000:2.0:1787829135.554014:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829135.554017:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829135.554022:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829135.554039:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829135/real 1787829135] req@ffff9e6804421e00 x1874668790638720/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829135.554043:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790639104/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829135.554066:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829135.554068:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829135.554071:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829135.554072:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790638720/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829135.554077:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5772 00000100:00080000:2.0:1787829135.554082:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829135.554087:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787829135.554090:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787829135.554091:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829135.554108:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829135.554112:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790639232/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829135.554113:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790638720:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829135.554128:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829135.554130:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829135.554131:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829135.554135:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5772 00000100:00080000:2.0:1787829135.554137:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829135.554140:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829135.554143:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790638848:192.168.203.109@tcp:38:kworker.0 00000008:00100000:2.0:1787829135.554145:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829135.554156:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790639360/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829135.554168:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829135.554170:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829135.554172:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829135.554174:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5772 00000100:00080000:2.0:1787829135.554177:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829135.554182:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787829135.554183:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:2.0:1787829135.554188:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829135.554198:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829135/real 1787829135] req@ffff9e6804420b40 x1874668790638848/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829135.554200:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790639488/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829135.554229:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829135.554233:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790638848/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829135.554234:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829135.554237:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829135.554246:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5772 00000100:00080000:0.0:1787829135.554250:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829135.554252:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829135.554255:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829135.554262:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829135.554263:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790638848:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829135.554274:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790638976:192.168.203.109@tcp:38:kworker.0 00000008:00100000:2.0:1787829135.554275:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829135.554293:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790639616/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829135.554301:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829135.554312:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829135.554313:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829135/real 1787829135] req@ffff9e68044230c0 x1874668790638976/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829135.554322:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829135.554328:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829135.554332:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790638976/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829135.554337:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5772 00000100:00080000:2.0:1787829135.554343:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829135.554349:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829135.554350:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787829135.554358:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:2.0:1787829135.554361:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829135.554362:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790638976:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829135.554371:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790639104:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829135.554383:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790639744/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829135.554390:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829135.554400:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829135/real 1787829135] req@ffff9e68044212c0 x1874668790639104/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829135.554408:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790639104/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829135.554415:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829135.554417:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829135.554419:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790639104:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829135.554423:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790639232:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829135.554432:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829135.554438:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829135/real 1787829135] req@ffff9e6804423840 x1874668790639232/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829135.554446:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790639232/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829135.554451:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829135.554453:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829135.554455:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790639232:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829135.554466:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790639360:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829135.554476:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829135.554482:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829135/real 1787829135] req@ffff9e68044221c0 x1874668790639360/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829135.554489:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790639360/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829135.554496:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829135.554497:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829135.554499:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790639360:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829135.554503:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790639488:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829135.554512:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829135.554517:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829135/real 1787829135] req@ffff9e6804420780 x1874668790639488/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829135.554524:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790639488/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829135.554534:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829135.554536:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829135.554538:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790639488:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829135.554541:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790639616:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829135.554550:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829135.554556:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829135/real 1787829135] req@ffff9e6804422d00 x1874668790639616/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829135.554567:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790639616/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829135.554576:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829135.554579:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829135.554582:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790639616:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829135.554589:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790639744:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829135.554608:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829135.554619:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829135/real 1787829135] req@ffff9e6804422940 x1874668790639744/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829135.554627:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790639744/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829135.554632:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829135.554634:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829135.554637:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790639744:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829136.065800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829136.257846:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829137.089772:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829137.281861:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829138.113817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829138.305850:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829139.137757:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829139.329844:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829140.161779:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829140.353819:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829140.673799:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829140.673807:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829140.673811:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829140.673819:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5777 00000100:00080000:3.0:1787829140.673827:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829140.673833:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829140.673863:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790639872/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829140.673885:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829140.673887:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829140.673888:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829140.673891:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5777 00000100:00080000:3.0:1787829140.673893:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829140.673896:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829140.673904:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829140.673912:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790640000/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829140.673919:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829140.673922:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829140.673923:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829140.673926:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5777 00000100:00080000:3.0:1787829140.673927:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829140.673930:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829140.673934:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829140.673935:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790639872:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829140.673942:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790640128/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829140.673949:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829140.673952:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829140.673954:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829140.673956:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5777 00000100:00080000:3.0:1787829140.673958:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829140.673960:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829140.673963:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829140.673971:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790640256/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829140.673977:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829140.673979:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829140.673981:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829140.673984:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5777 00000100:00080000:3.0:1787829140.673986:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829140.673988:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829140.673991:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829140.673992:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829140.674001:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790640384/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829140.674008:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829140.674010:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829140.674012:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829140.674014:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5777 00000100:00080000:3.0:1787829140.674016:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829140.674019:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829140.674022:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829140.674039:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829140/real 1787829140] req@ffff9e680a2d43c0 x1874668790639872/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829140.674042:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790640512/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829140.674070:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829140.674072:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790639872/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829140.674073:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829140.674077:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829140.674084:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5777 00000100:00080000:3.0:1787829140.674089:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829140.674090:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829140.674092:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787829140.674098:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:3.0:1787829140.674098:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829140.674105:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790639872:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829140.674109:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790640640/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829140.674115:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790640000:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829140.674123:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829140.674126:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829140.674127:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829140.674131:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5777 00000100:00080000:3.0:1787829140.674134:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829140.674136:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829140.674140:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829140.674143:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829140.674152:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790640768/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829140.674157:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829140/real 1787829140] req@ffff9e680a2d52c0 x1874668790640000/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829140.674163:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829140.674169:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829140.674172:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829140.674179:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5777 00000100:00080000:3.0:1787829140.674182:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829140.674184:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790640000/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829140.674186:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829140.674192:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829140.674201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829140.674205:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790640896/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829140.674208:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829140.674215:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790640000:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829140.674242:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790640128:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829140.674263:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829140.674277:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829140/real 1787829140] req@ffff9e680a2d4000 x1874668790640128/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829140.674300:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790640128/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829140.674319:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829140.674322:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829140.674327:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790640128:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829140.674334:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790640256:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829140.674354:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829140.674364:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829140/real 1787829140] req@ffff9e680a2d4780 x1874668790640256/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829140.674377:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790640256/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829140.674385:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829140.674388:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829140.674390:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790640256:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829140.674394:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790640384:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829140.674406:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829140.674413:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829140/real 1787829140] req@ffff9e680a2d5a40 x1874668790640384/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829140.674420:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790640384/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829140.674426:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829140.674428:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829140.674430:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790640384:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829140.674433:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790640512:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829140.674443:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829140.674448:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829140/real 1787829140] req@ffff9e680a2d6580 x1874668790640512/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829140.674455:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790640512/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829140.674462:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829140.674463:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829140.674465:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790640512:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829140.674476:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790640640:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829140.674488:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829140.674499:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829140/real 1787829140] req@ffff9e680a2d70c0 x1874668790640640/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829140.674511:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790640640/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829140.674520:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829140.674523:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829140.674527:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790640640:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829140.674533:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790640768:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829140.674554:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829140.674564:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829140/real 1787829140] req@ffff9e680a2d7c00 x1874668790640768/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829140.674577:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790640768/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829140.674586:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829140.674589:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829140.674593:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790640768:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829140.674599:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790640896:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829140.674615:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829140.674654:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829140/real 1787829140] req@ffff9e680a2d4b40 x1874668790640896/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829140.674668:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790640896/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829140.674676:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829140.674690:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829140.674717:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790640896:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829141.185817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829141.377849:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829142.209776:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829142.401826:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829143.233808:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829143.425845:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829144.257774:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829144.449810:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829145.281801:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829145.473851:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829145.793775:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829145.793783:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829145.793787:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829145.793795:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5782 00000100:00080000:3.0:1787829145.793804:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829145.793811:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829145.793843:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790641024/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829145.793862:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829145.793865:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829145.793866:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829145.793869:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5782 00000100:00080000:3.0:1787829145.793871:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829145.793874:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829145.793880:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829145.793889:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790641152/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829145.793906:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829145.793910:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790641024:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829145.793911:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829145.793914:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829145.793922:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5782 00000100:00080000:3.0:1787829145.793925:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829145.793928:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829145.793931:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829145.793941:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790641280/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829145.793947:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829145.793950:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829145.793951:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829145.793953:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5782 00000100:00080000:3.0:1787829145.793955:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829145.793957:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829145.793960:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829145.793966:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790641408/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829145.793972:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829145.793975:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829145.793976:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829145.793978:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5782 00000100:00080000:3.0:1787829145.793980:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829145.793984:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829145.793990:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829145.793994:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829145.794003:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790641536/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829145.794009:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829145.794012:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829145.794013:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829145.794016:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5782 00000100:00080000:3.0:1787829145.794019:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829145.794021:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829145/real 1787829145] req@ffff9e680a2d7480 x1874668790641024/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829145.794025:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829145.794031:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829145.794045:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790641664/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829145.794048:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790641024/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829145.794065:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829145.794069:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829145.794071:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829145.794071:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829145.794081:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829145.794082:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5782 00000100:00080000:3.0:1787829145.794088:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829145.794091:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790641024:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829145.794094:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829145.794101:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829145.794108:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790641792/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829145.794115:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829145.794117:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787829145.794120:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790641152:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829145.794121:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829145.794130:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5782 00000100:00080000:3.0:1787829145.794133:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829145.794136:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829145.794140:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829145.794146:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829145.794150:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790641920/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829145.794157:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829145/real 1787829145] req@ffff9e680a2d5e00 x1874668790641152/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829145.794159:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829145.794167:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829145.794170:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829145.794176:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5782 00000100:00100000:0.0:1787829145.794180:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790641152/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829145.794180:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829145.794184:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829145.794190:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829145.794195:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829145.794201:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829145.794201:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790642048/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829145.794207:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790641152:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829145.794218:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790641280:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829145.794236:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829145.794242:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829145/real 1787829145] req@ffff9e680a2d6d00 x1874668790641280/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829145.794250:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790641280/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829145.794255:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829145.794256:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829145.794258:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790641280:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829145.794262:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790641408:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829145.794272:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829145.794277:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829145/real 1787829145] req@ffff9e680a2d61c0 x1874668790641408/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829145.794284:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790641408/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829145.794289:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829145.794291:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829145.794293:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790641408:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829145.794296:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790641536:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829145.794305:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829145.794310:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829145/real 1787829145] req@ffff9e680a2d4f00 x1874668790641536/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829145.794322:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790641536/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829145.794327:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829145.794329:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829145.794331:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790641536:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829145.794334:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790641664:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829145.794343:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829145.794349:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829145/real 1787829145] req@ffff9e680a2d4b40 x1874668790641664/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829145.794356:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790641664/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829145.794361:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829145.794363:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829145.794365:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790641664:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829145.794368:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790641792:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829145.794377:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829145.794382:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829145/real 1787829145] req@ffff9e680a2d7c00 x1874668790641792/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829145.794389:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790641792/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829145.794398:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829145.794400:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829145.794402:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790641792:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829145.794415:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790641920:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829145.794425:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829145.794430:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829145/real 1787829145] req@ffff9e680a2d70c0 x1874668790641920/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829145.794437:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790641920/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829145.794443:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829145.794444:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829145.794446:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790641920:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829145.794449:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790642048:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829145.794458:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829145.794463:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829145/real 1787829145] req@ffff9e680a2d6580 x1874668790642048/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829145.794470:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790642048/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829145.794474:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829145.794476:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829145.794479:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790642048:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829146.305794:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829146.497870:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829147.329809:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829147.521835:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829148.353796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829148.545865:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829149.377800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829149.569841:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829150.401850:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829150.593881:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829150.913823:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829150.913836:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829150.913841:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829150.913854:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5787 00000100:00080000:1.0:1787829150.913868:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829150.913878:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829150.913928:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790642176/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829150.913961:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829150.913965:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829150.913967:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829150.913972:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5787 00000100:00080000:1.0:1787829150.913975:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829150.913981:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829150.913997:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829150.914010:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790642304/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829150.914020:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829150.914024:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829150.914026:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829150.914031:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5787 00000100:00080000:1.0:1787829150.914035:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829150.914039:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829150.914045:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829150.914046:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790642176:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787829150.914059:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790642432/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829150.914071:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829150.914075:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829150.914077:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829150.914081:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5787 00000100:00080000:1.0:1787829150.914085:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829150.914089:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829150.914095:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829150.914106:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790642560/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829150.914116:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829150.914120:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1787829150.914124:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829150.914125:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829150.914131:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5787 00000100:00080000:1.0:1787829150.914136:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829150.914140:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829150.914145:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829150.914157:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790642688/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829150.914168:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829150.914173:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829150/real 1787829150] req@ffff9e6831aaa1c0 x1874668790642176/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829150.914174:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829150.914178:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829150.914188:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5787 00000100:00080000:1.0:1787829150.914195:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829150.914199:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790642176/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829150.914202:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829150.914212:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829150.914216:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829150.914221:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829150.914226:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790642176:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787829150.914230:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790642816/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829150.914236:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790642304:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829150.914249:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829150.914254:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829150.914256:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829150.914261:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829150.914265:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5787 00000100:00080000:1.0:1787829150.914271:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829150.914275:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829150/real 1787829150] req@ffff9e6831aa8780 x1874668790642304/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829150.914279:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829150.914290:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829150.914293:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790642304/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829150.914303:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787829150.914306:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790642944/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829150.914308:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829150.914315:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790642304:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829150.914328:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829150.914331:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829150.914333:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829150.914338:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5787 00000100:00080000:1.0:1787829150.914343:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829150.914347:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790642432:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829150.914351:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829150.914359:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829150.914368:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829150.914374:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790643072/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829150.914379:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829150/real 1787829150] req@ffff9e6831aab480 x1874668790642432/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829150.914403:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829150.914406:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790642432/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829150.914410:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829150.914414:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829150.914418:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829150.914423:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5787 00000100:00080000:0.0:1787829150.914425:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829150.914431:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829150.914432:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790642432:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829150.914441:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829150.914442:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790642560:192.168.203.109@tcp:8:kworker.0 00000008:00100000:1.0:1787829150.914450:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829150.914465:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829150.914467:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790643200/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829150.914476:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829150/real 1787829150] req@ffff9e6831aaad00 x1874668790642560/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829150.914490:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790642560/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829150.914496:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829150.914498:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829150.914501:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790642560:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829150.914505:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790642688:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829150.914515:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829150.914520:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829150/real 1787829150] req@ffff9e6831aa8000 x1874668790642688/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829150.914528:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790642688/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829150.914533:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829150.914535:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829150.914537:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790642688:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829150.914540:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790642816:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829150.914550:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829150.914557:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829150/real 1787829150] req@ffff9e6831aa9a40 x1874668790642816/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829150.914564:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790642816/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829150.914571:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829150.914572:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829150.914574:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790642816:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829150.914577:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790642944:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829150.914587:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829150.914598:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829150/real 1787829150] req@ffff9e6831aabc00 x1874668790642944/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829150.914606:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790642944/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829150.914632:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829150.914634:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829150.914636:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790642944:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829150.914648:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790643072:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829150.914661:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829150.914668:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829150/real 1787829150] req@ffff9e6831aaa940 x1874668790643072/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829150.914675:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790643072/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829150.914681:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829150.914683:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829150.914684:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790643072:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829150.914688:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790643200:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829150.914736:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829150.914744:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829150/real 1787829150] req@ffff9e6831aa8b40 x1874668790643200/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829150.914752:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790643200/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829175 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829150.914757:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829150.914759:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829150.914762:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790643200:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829151.425841:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829151.617898:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829152.449789:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829152.641877:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829153.473829:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829153.665965:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829154.497777:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829154.689871:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829155.521782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829155.713841:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829156.033783:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829156.033794:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829156.033799:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829156.033809:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5792 00000100:00080000:1.0:1787829156.033820:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829156.033827:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829156.033858:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790643328/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829156.033879:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829156.033882:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829156.033884:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829156.033887:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5792 00000100:00080000:1.0:1787829156.033890:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829156.033893:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829156.033901:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829156.033915:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790643328:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787829156.033916:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790643456/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829156.033930:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829156.033932:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829156.033933:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829156.033936:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5792 00000100:00080000:1.0:1787829156.033938:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829156.033948:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829156.033957:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829156.033970:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790643584/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829156.033987:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829156.033993:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1787829156.033994:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829156.033997:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829156.034003:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5792 00000100:00080000:1.0:1787829156.034008:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829156.034011:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829156.034016:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829156.034028:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790643712/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829156.034035:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829156.034037:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829156.034041:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829156.034042:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829156/real 1787829156] req@ffff9e6831aa9e00 x1874668790643328/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829156.034046:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5792 00000100:00080000:1.0:1787829156.034052:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829156.034058:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829156.034062:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829156.034068:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790643328/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829156.034076:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790643840/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829156.034080:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829156.034088:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829156.034095:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790643328:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829156.034098:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829156.034103:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829156.034106:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829156.034111:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5792 00000100:00080000:1.0:1787829156.034116:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829156.034117:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790643456:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829156.034125:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829156.034133:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829156.034144:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829156.034147:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790643968/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829156.034156:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829156/real 1787829156] req@ffff9e6831aa8f00 x1874668790643456/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829156.034158:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829156.034165:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829156.034168:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829156.034176:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5792 00000100:00100000:0.0:1787829156.034180:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790643456/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829156.034189:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829156.034190:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829156.034197:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787829156.034198:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829156.034204:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790643456:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787829156.034206:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829156.034211:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790643584:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829156.034217:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790644096/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829156.034225:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829156.034230:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829156.034231:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829156.034236:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829156.034241:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5792 00000100:00080000:1.0:1787829156.034244:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829156.034245:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829156/real 1787829156] req@ffff9e6831aaa580 x1874668790643584/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829156.034249:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829156.034256:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829156.034262:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790643584/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829156.034271:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829156.034276:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829156.034277:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790644224/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829156.034280:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790643584:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829156.034295:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790643712:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787829156.034300:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829156.034304:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829156.034305:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829156.034309:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5792 00000100:00080000:1.0:1787829156.034312:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829156.034314:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829156.034318:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829156.034330:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829156.034330:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790644352/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829156.034351:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829156/real 1787829156] req@ffff9e6831aa9680 x1874668790643712/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829156.034362:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790643712/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829156.034368:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829156.034370:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829156.034373:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790643712:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829156.034377:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790643840:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829156.034391:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829156.034398:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829156/real 1787829156] req@ffff9e6831aa8b40 x1874668790643840/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829156.034405:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790643840/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829156.034411:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829156.034412:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829156.034415:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790643840:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829156.034437:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790643968:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829156.034449:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829156.034455:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829156/real 1787829156] req@ffff9e6831aaa940 x1874668790643968/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829156.034463:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790643968/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829156.034469:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829156.034471:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829156.034473:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790643968:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829156.034477:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790644096:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829156.034487:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829156.034493:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829156/real 1787829156] req@ffff9e6831aabc00 x1874668790644096/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829156.034500:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790644096/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829156.034504:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829156.034506:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829156.034509:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790644096:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829156.034512:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790644224:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829156.034521:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829156.034526:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829156/real 1787829156] req@ffff9e6831aa9a40 x1874668790644224/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829156.034533:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790644224/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829156.034537:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829156.034539:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829156.034541:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790644224:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829156.034544:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790644352:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829156.034553:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829156.034559:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829156/real 1787829156] req@ffff9e6831aa8000 x1874668790644352/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829156.034565:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790644352/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829156.034570:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829156.034587:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829156.034589:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790644352:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829156.545782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829156.737900:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829157.569779:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829157.761911:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829158.593804:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829158.785860:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829159.617802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829159.809871:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829160.641799:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829160.833806:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829161.153815:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829161.153824:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829161.153828:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829161.153838:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5797 00000100:00080000:0.0:1787829161.153850:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829161.153857:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829161.153909:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790644480/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829161.153937:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829161.153951:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829161.153953:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829161.153957:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5797 00000100:00080000:0.0:1787829161.153961:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829161.153965:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829161.153974:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829161.153988:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790644608/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829161.153997:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790644480:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829161.154000:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829161.154007:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829161.154009:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829161.154014:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5797 00000100:00080000:0.0:1787829161.154016:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829161.154019:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829161.154023:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829161.154032:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790644736/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829161.154044:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829161.154047:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829161.154049:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829161.154053:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5797 00000100:00080000:0.0:1787829161.154138:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829161.154143:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829161.154147:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829161.154157:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790644864/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829161.154165:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829161.154168:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829161.154169:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829161.154172:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5797 00000100:00080000:0.0:1787829161.154175:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829161.154177:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829161.154181:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829161.154189:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790644992/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829161.154195:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829161.154198:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829161.154199:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829161.154201:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5797 00000100:00080000:0.0:1787829161.154204:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829161.154206:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829161.154209:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829161.154215:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790645120/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829161.154220:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829161.154222:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829161.154223:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829161.154225:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5797 00000100:00080000:0.0:1787829161.154228:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829161.154230:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829161.154232:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829161.154238:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790645248/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829161.154243:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829161.154245:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829161.154246:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829161.154248:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5797 00000100:00080000:0.0:1787829161.154250:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829161.154252:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829161.154255:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829161.154261:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790645376/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829161.154267:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829161.154277:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829161.154280:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829161.154281:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829161.154285:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5797 00000100:00080000:0.0:1787829161.154291:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829161.154294:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829161.154298:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829161.154306:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790645504/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829161.154363:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829161/real 1787829161] req@ffff9e6803e9b480 x1874668790644480/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829161.154386:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790644480/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829161.154397:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829161.154403:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829161.154407:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790644480:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829161.154431:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790644608:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829161.154459:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829161.154474:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829161/real 1787829161] req@ffff9e6803e99e00 x1874668790644608/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829161.154488:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790644608/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829161.154504:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829161.154508:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829161.154512:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790644608:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829161.154519:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790644736:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829161.154564:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829161.154577:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829161/real 1787829161] req@ffff9e6803e99680 x1874668790644736/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829161.154590:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790644736/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829161.154599:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829161.154603:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829161.154606:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790644736:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829161.154613:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790644864:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829161.154644:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829161.154658:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829161/real 1787829161] req@ffff9e6803e9a580 x1874668790644864/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829161.154672:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790644864/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829161.154691:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829161.154721:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829161.154727:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790644864:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829161.154735:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790644992:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829161.154780:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829161.154795:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829161/real 1787829161] req@ffff9e6803e9b840 x1874668790644992/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829161.154817:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790644992/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829161.154827:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829161.154831:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829161.154834:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790644992:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829161.154841:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790645120:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829161.154863:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829161.154877:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829161/real 1787829161] req@ffff9e6803e9a1c0 x1874668790645120/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829161.154889:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790645120/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829161.154899:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829161.154903:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829161.154906:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790645120:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829161.154913:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790645248:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829161.154933:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829161.154953:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829161/real 1787829161] req@ffff9e6803e98780 x1874668790645248/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829161.154967:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790645248/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829161.154975:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829161.154978:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829161.154981:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790645248:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829161.154988:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790645376:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829161.155009:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829161.155022:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829161/real 1787829161] req@ffff9e6803e98b40 x1874668790645376/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829161.155035:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790645376/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829161.155043:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829161.155046:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829161.155050:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790645376:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829161.155056:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790645504:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829161.155077:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829161.155089:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829161/real 1787829161] req@ffff9e6803e9a940 x1874668790645504/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829161.155101:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790645504/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829161.155110:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829161.155113:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829161.155117:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790645504:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829161.665791:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829161.857915:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829162.689891:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829162.881915:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829163.713769:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829163.905842:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829164.737808:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829164.929842:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829165.761811:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829165.953860:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829166.273798:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829166.273806:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829166.273811:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829166.273818:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5802 00000100:00080000:3.0:1787829166.273827:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829166.273833:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829166.273871:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790645632/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829166.273892:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829166.273894:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829166.273896:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829166.273898:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5802 00000100:00080000:3.0:1787829166.273902:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829166.273905:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829166.273913:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829166.273924:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790645760/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829166.273925:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790645632:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829166.273944:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829166.273948:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829166.273951:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829166.273956:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5802 00000100:00080000:3.0:1787829166.273961:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829166.273965:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829166.273971:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829166.273983:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790645888/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829166.273993:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829166.273997:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829166.273999:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829166.274004:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5802 00000100:00080000:3.0:1787829166.274008:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829166.274013:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829166.274019:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829166.274026:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790645760:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829166.274036:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790646016/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829166.274053:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829166.274058:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790645888:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829166.274058:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829166.274061:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829166.274067:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5802 00000100:00080000:3.0:1787829166.274072:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829166.274075:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829166.274080:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829166.274086:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790646016:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829166.274092:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790646144/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829166.274101:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829166.274104:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829166.274105:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829166.274108:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5802 00000100:00080000:3.0:1787829166.274112:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829166.274115:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829166.274119:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829166.274127:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790646144:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829166.274130:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790646272/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829166.274142:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829166.274145:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829166.274146:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829166.274149:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5802 00000100:00080000:3.0:1787829166.274153:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829166.274155:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829166.274158:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829166.274163:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790646272:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829166.274168:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790646400/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829166.274177:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829166.274179:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829166.274181:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829166.274186:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5802 00000100:00100000:0.0:1787829166.274187:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790646400:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829166.274189:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829166.274194:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829166.274197:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829166.274207:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790646528/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829166.274220:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829166.274223:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829166.274224:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829166.274227:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5802 00000100:00080000:3.0:1787829166.274229:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829166.274232:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829166.274236:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829166.274244:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790646528:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829166.274245:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790646656/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829166.274285:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790646656:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829166.785822:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829166.977891:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829167.809801:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829168.001881:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829168.833818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829169.025870:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787829169.345945:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:2.0:1787829169.345997:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787829169.346017:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787829169.346069:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:2.0:1787829169.346081:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787829169.346089:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787829169.346099:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787829169.346107:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787829169.346113:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787829169.346126:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787829169.346141:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1787829169.346186:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829166/real 1787829169] req@ffff9e680a2d4b40 x1874668790646656/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829169.346249:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790646656/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829169.346273:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829169.346283:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829169.346292:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790646656:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829169.346309:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829166/real 1787829169] req@ffff9e680a2d7c00 x1874668790646528/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829169.346344:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790646528/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829169.346361:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829169.346368:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829169.346375:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790646528:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829169.346388:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829166/real 1787829169] req@ffff9e680a2d70c0 x1874668790646400/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829169.346411:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790646400/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829169.346427:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829169.346433:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829169.346439:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790646400:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829169.346477:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829166/real 1787829169] req@ffff9e680a2d6580 x1874668790646272/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829169.346525:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790646272/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829169.346545:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829169.346552:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829169.346557:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790646272:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829169.346571:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829166/real 1787829169] req@ffff9e680a2d43c0 x1874668790646144/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829169.346593:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790646144/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829169.346607:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829169.346613:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829169.346619:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790646144:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829169.346632:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829166/real 1787829169] req@ffff9e680a2d52c0 x1874668790646016/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829169.346656:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790646016/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829169.346673:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829169.346680:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829169.346686:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790646016:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829169.346750:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829166/real 1787829169] req@ffff9e680a2d4780 x1874668790645760/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829169.346777:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790645760/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829169.346794:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829169.346801:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829169.346809:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790645760:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829169.346823:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829166/real 1787829169] req@ffff9e680a2d4000 x1874668790645888/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829169.346845:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790645888/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829169.346861:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829169.346868:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829169.346875:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790645888:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829169.346892:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829166/real 1787829169] req@ffff9e680a2d5a40 x1874668790645632/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829169.346920:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790645632/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829169.346937:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829169.346945:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829169.346951:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790645632:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829169.857793:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829170.049917:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829170.881836:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829171.073910:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829171.393819:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829171.393827:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829171.393831:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829171.393842:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5808 00000100:00080000:2.0:1787829171.393852:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829171.393858:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829171.393890:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790646784/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829171.393913:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829171.393915:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829171.393917:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829171.393920:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5808 00000100:00080000:2.0:1787829171.393922:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829171.393925:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829171.393932:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829171.393941:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790646912/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829171.393952:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790646784:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829171.393955:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829171.393962:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829171.393964:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829171.393967:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5808 00000100:00080000:2.0:1787829171.393969:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829171.393971:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829171.393974:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829171.393985:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790647040/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829171.393994:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829171.393997:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829171.393999:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829171.394003:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5808 00000100:00080000:2.0:1787829171.394007:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829171.394011:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829171.394018:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829171.394031:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829171.394032:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790647168/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829171.394055:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829171.394059:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829171.394061:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829171.394065:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5808 00000100:00100000:0.0:1787829171.394068:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829171/real 1787829171] req@ffff9e6804423480 x1874668790646784/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829171.394081:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829171.394090:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829171.394096:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829171.394102:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790646784/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829171.394112:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790647296/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829171.394116:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829171.394126:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829171.394131:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829171.394145:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790646784:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829171.394150:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829171.394155:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829171.394160:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5808 00000100:00080000:2.0:1787829171.394165:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829171.394171:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829171.394176:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829171.394177:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790646912:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829171.394194:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790647424/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829171.394205:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829171.394209:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:0.0:1787829171.394212:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829171.394213:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829171.394222:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5808 00000100:00080000:2.0:1787829171.394227:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829171.394233:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829171.394235:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829171/real 1787829171] req@ffff9e6804421680 x1874668790646912/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829171.394240:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829171.394257:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790647552/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829171.394261:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790646912/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829171.394300:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829171.394303:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829171.394309:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829171.394309:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787829171.394317:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790646912:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829171.394320:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829171.394327:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5808 00000100:00080000:2.0:1787829171.394333:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829171.394345:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790647040:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829171.394353:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829171.394359:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829171.394373:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790647680/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829171.394383:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829171.394388:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829171.394394:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829171.394396:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829171.394404:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829171/real 1787829171] req@ffff9e6804422580 x1874668790647040/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829171.394404:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5808 00000100:00080000:2.0:1787829171.394410:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829171.394418:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829171.394428:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829171.394434:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790647040/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829171.394444:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790647808/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829171.394449:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829171.394460:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829171.394481:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790647040:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829171.394489:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790647168:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829171.394531:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829171.394544:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829171/real 1787829171] req@ffff9e6804420f00 x1874668790647168/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829171.394557:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790647168/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829171.394564:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829171.394568:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829171.394571:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790647168:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829171.394577:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790647296:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829171.394596:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829171.394607:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829171/real 1787829171] req@ffff9e6804422940 x1874668790647296/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829171.394618:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790647296/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829171.394632:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829171.394635:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829171.394639:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790647296:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829171.394657:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790647424:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829171.394684:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829171.394725:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829171/real 1787829171] req@ffff9e6804422d00 x1874668790647424/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829171.394738:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790647424/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829171.394749:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829171.394754:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829171.394758:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790647424:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829171.394766:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790647552:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829171.394791:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829171.394804:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829171/real 1787829171] req@ffff9e6804420780 x1874668790647552/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829171.394817:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790647552/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829171.394824:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829171.394827:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829171.394831:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790647552:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829171.394837:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790647680:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829171.394857:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829171.394870:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829171/real 1787829171] req@ffff9e68044221c0 x1874668790647680/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829171.394881:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790647680/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829171.394889:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829171.394892:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829171.394895:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790647680:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829171.394901:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790647808:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829171.394921:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829171.394933:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829171/real 1787829171] req@ffff9e6804423840 x1874668790647808/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829171.394945:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790647808/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829171.394952:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829171.394955:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829171.394958:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790647808:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829171.905826:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829172.097991:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829172.929822:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829173.121868:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829173.953844:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829174.145866:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829174.977795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829175.169863:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829176.001787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829176.193886:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829176.513816:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829176.513828:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829176.513833:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829176.513845:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5813 00000100:00080000:2.0:1787829176.513859:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829176.513866:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829176.513916:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790647936/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829176.513956:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829176.513960:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829176.513963:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829176.513969:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5813 00000100:00080000:2.0:1787829176.513973:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829176.513978:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829176.513990:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829176.514009:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790648064/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829176.514020:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829176.514024:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829176.514027:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829176.514032:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790647936:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829176.514033:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5813 00000100:00080000:2.0:1787829176.514042:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829176.514052:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829176.514058:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829176.514084:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790648192/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829176.514092:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829176.514096:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829176.514097:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829176.514101:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5813 00000100:00080000:2.0:1787829176.514104:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829176.514106:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829176.514110:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829176.514123:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790648320/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829176.514138:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829176.514141:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829176.514145:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829176.514146:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829176.514163:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5813 00000100:00080000:2.0:1787829176.514167:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829176.514169:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829176.514172:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829176.514179:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790648448/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829176.514187:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829176.514191:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829176.514192:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829176.514196:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5813 00000100:00080000:2.0:1787829176.514199:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829176.514203:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829176.514208:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829176.514229:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790648576/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829176.514231:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829176/real 1787829176] req@ffff9e68044212c0 x1874668790647936/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829176.514283:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829176.514287:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829176.514289:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829176.514296:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5813 00000100:00080000:2.0:1787829176.514299:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829176.514310:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829176.514324:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829176.514337:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790648704/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829176.514348:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790647936/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829176.514351:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829176.514361:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829176.514364:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829176.514374:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5813 00000100:00080000:2.0:1787829176.514384:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829176.514394:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829176.514401:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829176.514404:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787829176.514412:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790648832/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829176.514413:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829176.514450:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790647936:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829176.514455:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829176.514467:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829176.514470:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829176.514477:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5813 00000100:00080000:2.0:1787829176.514485:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829176.514490:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829176.514496:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829176.514497:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790648064:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829176.514517:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790648960/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829176.514538:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829176.514557:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829176/real 1787829176] req@ffff9e68044230c0 x1874668790648064/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829176.514569:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790648064/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829176.514577:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829176.514580:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829176.514583:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790648064:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829176.514588:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790648192:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829176.514604:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829176.514611:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829176/real 1787829176] req@ffff9e6804420b40 x1874668790648192/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829176.514620:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790648192/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829176.514649:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829176.514661:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829176.514664:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790648192:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829176.514673:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790648320:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829176.514692:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829176.514761:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829176/real 1787829176] req@ffff9e6804421e00 x1874668790648320/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829176.514779:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790648320/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829176.514806:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829176.514813:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829176.514816:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790648320:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829176.514822:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790648448:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829176.514862:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829176.514872:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829176/real 1787829176] req@ffff9e6804423840 x1874668790648448/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829176.514882:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790648448/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829176.514888:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829176.514891:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829176.514911:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790648448:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829176.514917:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790648576:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829176.514935:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829176.514947:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829176/real 1787829176] req@ffff9e68044221c0 x1874668790648576/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829176.514961:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790648576/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829176.514974:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829176.514978:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829176.514982:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790648576:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829176.514990:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790648704:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829176.515012:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829176.515022:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829176/real 1787829176] req@ffff9e6804420780 x1874668790648704/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829176.515032:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790648704/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829176.515038:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829176.515040:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829176.515043:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790648704:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829176.515047:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790648832:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829176.515061:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829176.515074:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829176/real 1787829176] req@ffff9e6804422d00 x1874668790648832/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829176.515088:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790648832/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829176.515098:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829176.515102:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829176.515106:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790648832:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829176.515134:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790648960:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829176.515154:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829176.515163:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829176/real 1787829176] req@ffff9e6804422940 x1874668790648960/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829176.515172:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790648960/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829176.515179:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829176.515181:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829176.515183:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790648960:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829177.025801:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829177.217849:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829178.049817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829178.241873:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829179.073873:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829179.265879:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829180.097802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829180.289963:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829181.121809:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829181.313898:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829181.633818:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829181.633830:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829181.633835:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829181.633847:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5818 00000100:00080000:2.0:1787829181.633859:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829181.633867:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829181.633910:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790649088/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829181.633945:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829181.633949:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829181.633952:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829181.633957:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5818 00000100:00080000:2.0:1787829181.633962:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829181.633967:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829181.633980:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829181.633995:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790649088:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829181.633998:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790649216/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829181.634022:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829181.634026:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829181.634028:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829181.634033:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5818 00000100:00080000:2.0:1787829181.634038:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829181.634042:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829181.634047:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829181.634058:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790649344/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829181.634065:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829181.634068:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829181.634070:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829181.634073:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5818 00000100:00080000:2.0:1787829181.634077:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829181.634080:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829181.634083:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829181.634090:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829181.634094:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790649472/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829181.634102:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829181.634105:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829181.634106:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829181.634109:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5818 00000100:00080000:2.0:1787829181.634112:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829181.634115:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829181.634118:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829181.634127:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790649600/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829181.634132:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829181.634134:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829181.634136:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829181.634139:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5818 00000100:00080000:2.0:1787829181.634142:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829181.634145:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829181.634149:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829181.634153:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829181/real 1787829181] req@ffff9e6804420f00 x1874668790649088/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829181.634164:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790649728/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829181.634176:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829181.634178:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790649088/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829181.634184:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829181.634188:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829181.634197:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5818 00000100:00080000:0.0:1787829181.634201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829181.634202:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829181.634213:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829181.634213:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829181.634223:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829181.634224:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790649088:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829181.634241:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790649856/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829181.634257:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790649216:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829181.634264:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829181.634269:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829181.634272:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829181.634277:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5818 00000100:00080000:2.0:1787829181.634283:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829181.634285:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829181.634291:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829181.634299:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829181.634301:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829181/real 1787829181] req@ffff9e6804422580 x1874668790649216/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829181.634314:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790649216/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829181.634318:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790649984/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829181.634328:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829181.634336:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829181.634337:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829181.634345:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787829181.634346:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790649216:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829181.634348:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829181.634358:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5818 00000100:00100000:0.0:1787829181.634362:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790649344:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829181.634365:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829181.634374:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829181.634380:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829181.634392:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829181.634411:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790650112/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829181.634424:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829181/real 1787829181] req@ffff9e6804421680 x1874668790649344/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829181.634434:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790649344/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829181.634442:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829181.634445:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829181.634448:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790649344:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829181.634453:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790649472:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829181.634469:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829181.634475:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829181/real 1787829181] req@ffff9e6804423480 x1874668790649472/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829181.634482:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790649472/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829181.634488:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829181.634490:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829181.634492:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790649472:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829181.634496:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790649600:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829181.634506:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829181.634511:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829181/real 1787829181] req@ffff9e6804422940 x1874668790649600/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829181.634519:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790649600/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829181.634524:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829181.634525:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829181.634527:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790649600:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829181.634530:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790649728:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829181.634540:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829181.634547:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829181/real 1787829181] req@ffff9e6804422d00 x1874668790649728/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829181.634554:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790649728/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829181.634561:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829181.634563:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829181.634564:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790649728:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829181.634577:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790649856:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829181.634588:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829181.634594:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829181/real 1787829181] req@ffff9e6804420780 x1874668790649856/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829181.634602:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790649856/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829181.634607:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829181.634626:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829181.634628:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790649856:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829181.634632:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790649984:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829181.634644:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829181.634651:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829181/real 1787829181] req@ffff9e68044221c0 x1874668790649984/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829181.634659:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790649984/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829181.634670:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829181.634672:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829181.634690:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790649984:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829181.634693:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790650112:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829181.634704:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829181.634709:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829181/real 1787829181] req@ffff9e6804423840 x1874668790650112/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829181.634716:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790650112/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829181.634722:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829181.634751:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829181.634753:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790650112:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829182.145767:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829182.337860:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829183.169778:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829183.361871:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829184.193806:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829184.385863:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829185.217802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829185.409845:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829186.241780:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829186.433908:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829186.753803:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829186.753815:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829186.753819:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829186.753830:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5823 00000100:00080000:2.0:1787829186.753843:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829186.753850:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829186.753896:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790650240/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829186.753923:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829186.753926:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829186.753928:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829186.753931:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5823 00000100:00080000:2.0:1787829186.753933:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829186.753936:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829186.753945:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829186.753954:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790650368/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829186.753962:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829186.753965:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829186.753966:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829186.753970:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5823 00000100:00080000:2.0:1787829186.753971:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829186.753974:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829186.753977:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829186.753988:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790650496/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829186.753993:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790650240:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829186.753998:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829186.754010:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829186.754012:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829186.754016:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5823 00000100:00080000:2.0:1787829186.754019:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829186.754021:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829186.754043:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829186.754053:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790650624/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829186.754060:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829186.754063:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829186.754064:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829186.754070:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5823 00000100:00080000:2.0:1787829186.754074:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829186.754077:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829186.754080:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829186.754093:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790650752/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829186.754104:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829186.754112:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829186.754117:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829186.754120:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829186.754127:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5823 00000100:00080000:2.0:1787829186.754132:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829186.754140:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829186.754155:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829186/real 1787829186] req@ffff9e6804421e00 x1874668790650240/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829186.754159:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829186.754184:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790650880/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829186.754188:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790650240/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829186.754251:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829186.754252:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829186.754265:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829186.754268:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829186.754269:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829186.754276:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790650240:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829186.754277:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5823 00000100:00080000:2.0:1787829186.754288:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829186.754295:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790650368:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829186.754295:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829186.754310:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829186.754326:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790651008/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829186.754337:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829186.754349:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829186.754353:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829186/real 1787829186] req@ffff9e6804420b40 x1874668790650368/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829186.754355:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829186.754391:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829186.754404:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5823 00000100:00080000:2.0:1787829186.754419:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829186.754428:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829186.754434:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829186.754440:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790650368/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829186.754447:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790651136/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829186.754503:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829186.754504:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829186.754514:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829186.754514:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787829186.754520:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790650368:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829186.754521:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829186.754537:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5823 00000100:00080000:2.0:1787829186.754552:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829186.754557:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829186.754564:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829186.754575:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790650496:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829186.754579:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790651264/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829186.754605:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829186.754621:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829186/real 1787829186] req@ffff9e68044230c0 x1874668790650496/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829186.754638:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790650496/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829186.754644:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829186.754647:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829186.754666:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790650496:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829186.754671:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790650624:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829186.754687:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829186.754695:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829186/real 1787829186] req@ffff9e68044212c0 x1874668790650624/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829186.754703:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790650624/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829186.754748:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829186.754751:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829186.754754:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790650624:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829186.754758:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790650752:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829186.754772:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829186.754797:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829186/real 1787829186] req@ffff9e6804423840 x1874668790650752/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829186.754805:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790650752/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829186.754811:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829186.754814:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829186.754816:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790650752:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829186.754820:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790650880:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829186.754833:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829186.754839:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829186/real 1787829186] req@ffff9e68044221c0 x1874668790650880/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829186.754846:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790650880/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829186.754866:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829186.754869:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829186.754871:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790650880:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829186.754876:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790651008:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829186.754888:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829186.754894:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829186/real 1787829186] req@ffff9e6804420780 x1874668790651008/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829186.754918:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790651008/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829186.754924:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829186.754925:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829186.754927:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790651008:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829186.754931:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790651136:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829186.754942:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829186.754948:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829186/real 1787829186] req@ffff9e6804422d00 x1874668790651136/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829186.754955:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790651136/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829186.754960:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829186.754961:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829186.754964:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790651136:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829186.754978:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790651264:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829186.754988:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829186.754994:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829186/real 1787829186] req@ffff9e6804422940 x1874668790651264/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829186.755002:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790651264/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829186.755007:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829186.755009:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829186.755011:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790651264:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829187.265822:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829187.457898:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829188.289829:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829188.481871:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829189.313799:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829189.505918:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829190.337828:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829190.529854:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829191.361803:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829191.554027:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829191.873911:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829191.873928:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829191.873936:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829191.873969:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5828 00000100:00080000:3.0:1787829191.873988:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829191.873999:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829191.874087:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790651392/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829191.874138:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829191.874142:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829191.874144:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829191.874149:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5828 00000100:00080000:3.0:1787829191.874152:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829191.874156:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829191.874169:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829191.874207:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790651520/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829191.874223:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790651392:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829191.874229:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829191.874237:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829191.874241:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829191.874249:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5828 00000100:00080000:3.0:1787829191.874257:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829191.874263:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829191.874270:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829191.874286:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790651648/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829191.874297:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829191.874300:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829191.874302:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829191.874306:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5828 00000100:00080000:3.0:1787829191.874311:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829191.874315:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829191.874344:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829191.874346:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829191.874388:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790651776/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829191.874405:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829191.874410:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829191.874412:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829191.874418:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5828 00000100:00080000:3.0:1787829191.874424:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829191.874430:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829191.874437:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829191.874465:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790651904/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829191.874468:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829191/real 1787829191] req@ffff9e680a2d4f00 x1874668790651392/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829191.874494:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829191.874509:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829191.874515:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829191.874524:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5828 00000100:00080000:3.0:1787829191.874529:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829191.874535:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829191.874541:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829191.874544:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790651392/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829191.874561:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790652032/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829191.874566:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829191.874590:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829191.874593:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829191.874596:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829191.874601:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5828 00000100:00080000:0.0:1787829191.874604:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829191.874605:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829191.874611:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790651392:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829191.874612:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829191.874621:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829191.874631:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790652160/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829191.874662:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829191.874670:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829191.874677:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829191.874678:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790651520:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829191.874698:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5828 00000100:00080000:3.0:1787829191.874709:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829191.874718:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829191.874729:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829191.874807:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829191.874845:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829191/real 1787829191] req@ffff9e680a2d61c0 x1874668790651520/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829191.874845:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790652288/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829191.874904:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829191.874906:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790651520/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829191.874922:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829191.874931:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829191.874943:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829191.874944:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5828 00000100:00080000:0.0:1787829191.874976:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829191.874977:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829191.874984:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790651520:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829191.874993:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829191.875005:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790651648:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787829191.875005:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829191.875051:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790652416/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829191.875070:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829191.875105:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829191/real 1787829191] req@ffff9e680a2d6d00 x1874668790651648/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829191.875119:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790651648/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829191.875128:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829191.875132:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829191.875135:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790651648:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829191.875141:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790651776:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829191.875179:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829191.875196:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829191/real 1787829191] req@ffff9e680a2d5e00 x1874668790651776/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829191.875213:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790651776/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829191.875226:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829191.875231:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829191.875236:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790651776:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829191.875262:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790651904:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829191.875297:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829191.875313:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829191/real 1787829191] req@ffff9e680a2d7480 x1874668790651904/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829191.875371:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790651904/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829191.875389:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829191.875395:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829191.875402:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790651904:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829191.875414:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790652032:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829191.875455:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829191.875479:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829191/real 1787829191] req@ffff9e680a2d5a40 x1874668790652032/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829191.875500:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790652032/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829191.875513:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829191.875517:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829191.875523:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790652032:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829191.875531:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790652160:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829191.875559:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829191.875587:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829191/real 1787829191] req@ffff9e680a2d4000 x1874668790652160/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829191.875595:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790652160/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829191.875600:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829191.875602:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829191.875604:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790652160:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829191.875621:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790652288:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829191.875634:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829191.875664:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829191/real 1787829191] req@ffff9e680a2d4780 x1874668790652288/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829191.875678:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790652288/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829191.875689:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829191.875692:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829191.875696:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790652288:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829191.875703:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790652416:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829191.875723:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829191.875735:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829191/real 1787829191] req@ffff9e680a2d52c0 x1874668790652416/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829191.875794:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790652416/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829216 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829191.875802:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829191.875805:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829191.875808:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790652416:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829192.385801:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829192.577934:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829193.409781:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829193.601869:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829194.433848:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829194.625908:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829195.457790:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829195.649886:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829196.481876:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829196.673979:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829196.993788:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829196.993823:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829196.993829:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829196.993841:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5833 00000100:00080000:3.0:1787829196.993854:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829196.993863:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829196.993915:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790652544/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829196.993945:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829196.993949:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829196.993950:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829196.993954:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5833 00000100:00080000:3.0:1787829196.993957:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829196.993960:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829196.993970:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829196.993983:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790652672/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829196.993992:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829196.993995:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829196.993997:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829196.994000:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5833 00000100:00080000:3.0:1787829196.994002:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829196.994005:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829196.994008:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829196.994015:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790652800/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829196.994025:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829196.994028:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790652544:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829196.994029:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829196.994033:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829196.994041:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5833 00000100:00080000:3.0:1787829196.994044:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829196.994047:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829196.994051:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829196.994061:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790652928/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829196.994068:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829196.994070:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829196.994072:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829196.994076:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5833 00000100:00080000:3.0:1787829196.994079:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829196.994082:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829196.994086:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829196.994094:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790653056/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829196.994101:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829196.994103:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829196.994105:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829196.994108:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5833 00000100:00080000:3.0:1787829196.994112:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829196.994116:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829196.994119:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829196.994143:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829196.994146:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790653184/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829196.994160:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829196.994165:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829196.994168:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829196.994173:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5833 00000100:00080000:3.0:1787829196.994178:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829196.994184:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829196.994192:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829196.994202:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829196/real 1787829196] req@ffff9e680a2d43c0 x1874668790652544/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829196.994210:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790653312/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829196.994223:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829196.994227:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787829196.994230:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790652544/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829196.994231:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829196.994236:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5833 00000100:00080000:3.0:1787829196.994240:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829196.994247:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829196.994251:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829196.994264:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790653440/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829196.994269:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829196.994273:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829196.994277:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829196.994280:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829196.994281:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829196.994307:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790652544:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829196.994307:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5833 00000100:00080000:3.0:1787829196.994311:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829196.994319:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829196.994325:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829196.994328:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790652672:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829196.994351:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790653568/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829196.994377:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829196.994393:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829196/real 1787829196] req@ffff9e680a2d6580 x1874668790652672/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829196.994406:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790652672/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829196.994420:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829196.994423:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829196.994426:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790652672:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829196.994472:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790652800:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829196.994498:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829196.994518:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829196/real 1787829196] req@ffff9e680a2d70c0 x1874668790652800/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829196.994555:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790652800/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829196.994565:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829196.994570:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829196.994574:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790652800:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829196.994582:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790652928:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829196.994607:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829196.994622:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829196/real 1787829196] req@ffff9e680a2d7c00 x1874668790652928/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829196.994639:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790652928/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829196.994660:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829196.994664:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829196.994670:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790652928:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829196.994678:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790653056:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829196.994702:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829196.994716:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829196/real 1787829196] req@ffff9e680a2d4b40 x1874668790653056/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829196.994805:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790653056/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829196.994815:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829196.994817:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829196.994820:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790653056:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829196.994826:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790653184:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829196.994848:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829196.994862:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829196/real 1787829196] req@ffff9e680a2d52c0 x1874668790653184/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829196.994874:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790653184/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829196.994885:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829196.994888:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829196.994891:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790653184:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829196.994896:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790653312:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829196.994909:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829196.994920:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829196/real 1787829196] req@ffff9e680a2d4780 x1874668790653312/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829196.994929:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790653312/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829196.994935:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829196.994937:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829196.994939:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790653312:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829196.994944:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790653440:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829196.994958:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829196.994966:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829196/real 1787829196] req@ffff9e680a2d4000 x1874668790653440/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829196.994977:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790653440/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829196.994983:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829196.994985:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829196.994988:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790653440:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829196.994993:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790653568:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829196.995022:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829196.995049:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829196/real 1787829196] req@ffff9e680a2d5a40 x1874668790653568/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829196.995068:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790653568/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829221 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829196.995083:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829196.995090:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829196.995097:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790653568:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829197.505892:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829197.697935:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829198.529909:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829198.721921:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829199.553831:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829199.745838:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829200.577819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829200.769916:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829201.601819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829201.793840:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829202.113820:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829202.113828:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829202.113832:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829202.113842:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5838 00000100:00080000:2.0:1787829202.113851:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829202.113858:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829202.113891:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790653696/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829202.113912:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829202.113915:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829202.113916:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829202.113919:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5838 00000100:00080000:2.0:1787829202.113922:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829202.113925:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829202.113932:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829202.113940:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790653824/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829202.113954:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829202.113960:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787829202.113962:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790653696:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829202.113962:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829202.113973:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5838 00000100:00080000:2.0:1787829202.113982:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829202.113986:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829202.113989:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829202.114001:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790653952/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829202.114010:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829202.114013:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829202.114017:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829202.114021:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5838 00000100:00080000:2.0:1787829202.114026:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829202.114031:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829202.114037:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829202.114049:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790654080/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829202.114060:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829202.114068:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829202.114070:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829202.114074:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829202.114082:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5838 00000100:00080000:2.0:1787829202.114090:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829202.114105:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829202.114110:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829202.114127:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790654208/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829202.114135:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829202.114140:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829202/real 1787829202] req@ffff9e6804423480 x1874668790653696/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829202.114140:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829202.114144:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829202.114156:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5838 00000100:00080000:2.0:1787829202.114165:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829202.114173:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829202.114179:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829202.114191:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790653696/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829202.114192:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790654336/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829202.114267:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829202.114270:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829202.114272:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829202.114273:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829202.114279:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5838 00000100:00080000:2.0:1787829202.114283:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829202.114284:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829202.114291:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829202.114299:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790653696:192.168.203.109@tcp:250:kworker.0 00000008:00100000:2.0:1787829202.114301:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829202.114320:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790654464/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829202.114333:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829202.114337:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829202.114339:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829202.114342:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790653824:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829202.114350:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5838 00000100:00080000:2.0:1787829202.114358:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829202.114366:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829202.114372:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829202.114380:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829202.114394:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790654592/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829202.114403:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829202/real 1787829202] req@ffff9e6804421680 x1874668790653824/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829202.114406:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829202.114418:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829202.114425:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829202.114441:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5838 00000100:00080000:2.0:1787829202.114446:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829202.114452:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790653824/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829202.114452:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829202.114466:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829202.114486:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787829202.114490:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790654720/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829202.114506:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829202.114513:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790653824:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829202.114524:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790653952:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829202.114546:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829202.114557:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829202/real 1787829202] req@ffff9e6804422580 x1874668790653952/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829202.114569:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790653952/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829202.114577:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829202.114581:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829202.114585:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790653952:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829202.114592:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790654080:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829202.114611:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829202.114621:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829202/real 1787829202] req@ffff9e6804420f00 x1874668790654080/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829202.114633:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790654080/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829202.114642:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829202.114646:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829202.114649:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790654080:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829202.114655:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790654208:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829202.114669:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829202.114678:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829202/real 1787829202] req@ffff9e6804422940 x1874668790654208/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829202.114690:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790654208/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829202.114739:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829202.114743:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829202.114747:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790654208:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829202.114753:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790654336:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829202.114772:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829202.114782:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829202/real 1787829202] req@ffff9e6804422d00 x1874668790654336/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829202.114795:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790654336/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829202.114805:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829202.114809:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829202.114812:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790654336:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829202.114817:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790654464:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829202.114832:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829202.114841:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829202/real 1787829202] req@ffff9e6804420780 x1874668790654464/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829202.114853:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790654464/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829202.114869:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829202.114872:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829202.114876:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790654464:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829202.114896:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790654592:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829202.114912:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829202.114921:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829202/real 1787829202] req@ffff9e68044221c0 x1874668790654592/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829202.114932:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790654592/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829202.114942:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829202.114945:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829202.114948:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790654592:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829202.114954:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790654720:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829202.114969:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829202.114977:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829202/real 1787829202] req@ffff9e6804423840 x1874668790654720/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829202.114988:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790654720/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829202.114997:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829202.115000:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829202.115004:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790654720:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829202.625812:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829202.817851:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829203.649833:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829203.841844:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829204.673841:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829204.865854:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829205.697799:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829205.889849:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829206.721783:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829206.913886:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829207.233814:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829207.233826:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829207.233830:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829207.233841:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5843 00000100:00080000:2.0:1787829207.233854:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829207.233860:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829207.233893:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790654848/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829207.233916:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829207.233919:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829207.233920:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829207.233923:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5843 00000100:00080000:2.0:1787829207.233926:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829207.233928:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829207.233939:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829207.233949:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790654976/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829207.233956:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790654848:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829207.233959:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829207.233964:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829207.233965:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829207.233968:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5843 00000100:00080000:2.0:1787829207.233971:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829207.233973:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829207.233977:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829207.233984:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790655104/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829207.233991:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829207.233994:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829207.233996:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829207.233998:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5843 00000100:00080000:2.0:1787829207.234001:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829207.234006:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829207.234011:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829207.234016:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829207.234028:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790655232/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829207.234040:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829207.234044:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829207.234046:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829207.234051:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5843 00000100:00080000:2.0:1787829207.234055:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829207.234056:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829207/real 1787829207] req@ffff9e68044212c0 x1874668790654848/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829207.234062:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829207.234068:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829207.234084:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790655360/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829207.234090:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790654848/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829207.234093:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829207.234101:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829207.234103:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829207.234110:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5843 00000100:00080000:0.0:1787829207.234113:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829207.234113:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829207.234119:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787829207.234122:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:2.0:1787829207.234124:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829207.234128:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790654848:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829207.234136:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790655488/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829207.234143:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829207.234144:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829207.234146:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829207.234149:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5843 00000100:00100000:0.0:1787829207.234152:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790654976:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829207.234153:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829207.234159:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829207.234163:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829207.234171:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790655616/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829207.234179:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829207.234183:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829207.234184:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829207.234187:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829207.234191:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5843 00000100:00080000:2.0:1787829207.234195:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829207.234200:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829207.234201:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829207/real 1787829207] req@ffff9e68044230c0 x1874668790654976/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829207.234203:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829207.234230:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790655744/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829207.234234:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790654976/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829207.234250:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829207.234252:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829207.234253:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829207.234258:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829207.234260:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829207.234264:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5843 00000100:00100000:0.0:1787829207.234268:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790654976:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829207.234269:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829207.234276:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829207.234279:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790655104:192.168.203.109@tcp:38:kworker.0 00000008:00100000:2.0:1787829207.234282:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829207.234296:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790655872/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829207.234306:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829207.234317:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829207/real 1787829207] req@ffff9e6804420b40 x1874668790655104/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829207.234326:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790655104/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829207.234332:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829207.234334:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829207.234336:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790655104:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829207.234340:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790655232:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829207.234350:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829207.234356:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829207/real 1787829207] req@ffff9e6804421e00 x1874668790655232/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829207.234363:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790655232/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829207.234368:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829207.234370:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829207.234372:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790655232:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829207.234375:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790655360:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829207.234384:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829207.234390:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829207/real 1787829207] req@ffff9e6804423840 x1874668790655360/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829207.234397:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790655360/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829207.234403:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829207.234405:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829207.234406:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790655360:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829207.234410:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790655488:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829207.234419:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829207.234425:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829207/real 1787829207] req@ffff9e68044221c0 x1874668790655488/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829207.234431:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790655488/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829207.234438:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829207.234440:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829207.234441:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790655488:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829207.234453:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790655616:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829207.234463:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829207.234469:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829207/real 1787829207] req@ffff9e6804420780 x1874668790655616/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829207.234476:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790655616/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829207.234481:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829207.234482:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829207.234485:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790655616:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829207.234488:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790655744:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829207.234498:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829207.234504:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829207/real 1787829207] req@ffff9e6804422d00 x1874668790655744/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829207.234516:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790655744/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829207.234524:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829207.234527:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829207.234530:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790655744:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829207.234537:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790655872:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829207.234557:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829207.234569:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829207/real 1787829207] req@ffff9e6804422940 x1874668790655872/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829207.234582:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790655872/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829207.234590:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829207.234594:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829207.234597:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790655872:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829207.745805:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829207.937886:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829208.769821:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829208.961906:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829209.793890:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829209.985941:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829210.817798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829211.009903:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829211.841997:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829212.033896:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829212.353839:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829212.353853:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829212.353860:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829212.353871:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5849 00000100:00080000:0.0:1787829212.353883:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829212.353891:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829212.353943:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790656000/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829212.353977:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829212.353980:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829212.353982:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829212.353985:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5849 00000100:00080000:0.0:1787829212.353987:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829212.353990:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829212.354000:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829212.354011:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790656128/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829212.354020:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829212.354023:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829212.354025:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829212.354028:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5849 00000100:00080000:0.0:1787829212.354030:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829212.354033:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829212.354037:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829212.354046:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790656000:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787829212.354047:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790656256/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829212.354081:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829212.354084:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829212.354086:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829212.354090:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5849 00000100:00080000:0.0:1787829212.354093:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829212.354096:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829212.354100:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829212.354111:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790656384/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829212.354118:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829212.354121:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829212.354123:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829212.354125:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5849 00000100:00080000:0.0:1787829212.354128:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829212.354130:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829212.354133:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787829212.354139:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829212.354143:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790656512/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829212.354152:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829212.354156:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829212.354157:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829212.354160:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5849 00000100:00080000:0.0:1787829212.354167:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829212.354170:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829212.354174:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829212.354198:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790656640/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829212.354206:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829212.354209:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829212.354211:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829212.354218:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829212/real 1787829212] req@ffff9e6803e9ad00 x1874668790656000/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829212.354219:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5849 00000100:00080000:0.0:1787829212.354223:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829212.354230:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829212.354239:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829212.354256:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790656768/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829212.354267:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790656000/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829212.354268:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829212.354284:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829212.354288:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829212.354299:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5849 00000100:00080000:0.0:1787829212.354304:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829212.354308:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:3.0:1787829212.354310:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787829212.354314:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787829212.354321:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829212.354328:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790656000:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787829212.354332:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790656896/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829212.354347:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790656128:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829212.354368:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829212.354371:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829212.354373:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829212.354379:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5849 00000100:00080000:0.0:1787829212.354384:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829212.354388:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829212.354393:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787829212.354401:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829212.354411:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790657024/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829212.354421:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829212/real 1787829212] req@ffff9e6803e98f00 x1874668790656128/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829212.354433:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790656128/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829212.354447:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829212.354452:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829212.354456:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790656128:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829212.354483:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790656256:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829212.354508:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829212.354519:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829212/real 1787829212] req@ffff9e6803e992c0 x1874668790656256/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829212.354527:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790656256/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829212.354532:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829212.354535:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829212.354537:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790656256:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829212.354541:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790656384:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829212.354555:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829212.354561:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829212/real 1787829212] req@ffff9e6803e9bc00 x1874668790656384/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829212.354568:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790656384/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829212.354573:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829212.354575:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829212.354577:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790656384:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829212.354581:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790656512:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829212.354596:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829212.354606:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829212/real 1787829212] req@ffff9e6803e9a940 x1874668790656512/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829212.354618:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790656512/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829212.354626:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829212.354630:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829212.354633:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790656512:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829212.354639:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790656640:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829212.354659:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829212.354667:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829212/real 1787829212] req@ffff9e6803e98b40 x1874668790656640/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829212.354675:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790656640/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829212.354682:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829212.354684:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829212.354686:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790656640:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829212.354690:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790656768:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829212.354727:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829212.354739:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829212/real 1787829212] req@ffff9e6803e98780 x1874668790656768/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829212.354751:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790656768/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829212.354759:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829212.354762:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829212.354765:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790656768:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829212.354772:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790656896:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829212.354793:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829212.354804:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829212/real 1787829212] req@ffff9e6803e9a1c0 x1874668790656896/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829212.354814:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790656896/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829212.354829:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829212.354833:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829212.354836:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790656896:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829212.354844:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790657024:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829212.354864:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829212.354872:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829212/real 1787829212] req@ffff9e6803e9b840 x1874668790657024/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829212.354880:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790657024/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829212.354885:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829212.354888:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829212.354890:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790657024:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829212.865849:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829213.057843:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829213.889798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829214.081851:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829214.913776:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829215.105867:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829215.937788:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829216.129822:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829216.961815:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829217.153869:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829217.473780:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829217.473789:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829217.473793:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829217.473801:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5854 00000100:00080000:0.0:1787829217.473811:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829217.473817:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829217.473847:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790657152/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829217.473869:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829217.473873:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829217.473875:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829217.473878:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5854 00000100:00080000:0.0:1787829217.473882:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829217.473894:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829217.473905:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829217.473918:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790657280/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829217.473919:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790657152:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829217.473929:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829217.473932:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829217.473934:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829217.473937:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5854 00000100:00080000:0.0:1787829217.473941:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829217.473944:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829217.473948:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829217.473957:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790657408/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829217.473965:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829217.473968:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829217.473970:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829217.473973:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5854 00000100:00080000:0.0:1787829217.473976:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829217.473979:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:3.0:1787829217.473981:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787829217.473983:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829217.473992:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790657536/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829217.474000:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829217.474003:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829217.474005:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829217.474008:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5854 00000100:00080000:0.0:1787829217.474011:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829217.474014:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829217.474018:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829217/real 1787829217] req@ffff9e6803e9a580 x1874668790657152/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829217.474019:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829217.474030:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790657664/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829217.474034:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790657152/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829217.474039:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829217.474043:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829217.474044:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829217.474045:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829217.474048:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5854 00000100:00080000:3.0:1787829217.474060:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829217.474061:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829217.474064:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790657152:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829217.474065:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829217.474069:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829217.474077:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790657792/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829217.474085:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829217.474086:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790657280:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829217.474089:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829217.474090:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829217.474093:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5854 00000100:00080000:0.0:1787829217.474096:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829217.474100:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829217.474105:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829217.474107:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829217.474115:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790657920/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829217.474119:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829217/real 1787829217] req@ffff9e6803e99680 x1874668790657280/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829217.474125:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829217.474128:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829217.474130:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829217.474132:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790657280/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829217.474133:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5854 00000100:00080000:0.0:1787829217.474137:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829217.474141:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829217.474161:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787829217.474165:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829217.474169:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829217.474172:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790658048/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829217.474173:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790657280:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829217.474179:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790657408:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829217.474192:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829217.474195:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829217.474197:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829217.474201:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5854 00000100:00080000:0.0:1787829217.474204:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829217.474208:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:3.0:1787829217.474210:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787829217.474212:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829217.474222:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790658176/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829217.474222:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829217/real 1787829217] req@ffff9e6803e99e00 x1874668790657408/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829217.474242:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790657408/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829217.474251:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829217.474254:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829217.474257:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790657408:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829217.474264:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790657536:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829217.474284:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829217.474293:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829217/real 1787829217] req@ffff9e6803e9b480 x1874668790657536/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829217.474300:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790657536/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829217.474306:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829217.474308:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829217.474310:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790657536:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829217.474314:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790657664:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829217.474323:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829217.474329:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829217/real 1787829217] req@ffff9e6803e9b840 x1874668790657664/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829217.474336:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790657664/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829217.474355:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829217.474357:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829217.474359:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790657664:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829217.474371:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790657792:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829217.474382:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829217.474388:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829217/real 1787829217] req@ffff9e6803e9a1c0 x1874668790657792/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829217.474394:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790657792/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829217.474401:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829217.474403:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829217.474405:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790657792:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829217.474409:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790657920:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829217.474426:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829217.474437:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829217/real 1787829217] req@ffff9e6803e98780 x1874668790657920/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829217.474449:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790657920/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829217.474463:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829217.474466:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829217.474470:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790657920:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829217.474476:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790658048:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829217.474491:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829217.474498:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829217/real 1787829217] req@ffff9e6803e98b40 x1874668790658048/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829217.474506:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790658048/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829217.474511:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829217.474513:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829217.474515:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790658048:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829217.474518:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790658176:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829217.474528:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829217.474533:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829217/real 1787829217] req@ffff9e6803e9a940 x1874668790658176/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829217.474549:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790658176/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829217.474554:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829217.474556:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829217.474558:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790658176:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829217.985774:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829218.177817:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829219.009790:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829219.201861:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829220.033770:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829220.225851:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829221.057815:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829221.249826:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829222.081782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829222.273839:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829222.593828:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829222.593840:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829222.593844:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829222.593854:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5859 00000100:00080000:0.0:1787829222.593866:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829222.593874:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829222.593914:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790658304/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829222.593940:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829222.593942:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829222.593944:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829222.593947:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5859 00000100:00080000:0.0:1787829222.593949:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829222.593952:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829222.593959:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829222.593968:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790658432/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829222.593978:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829222.593980:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829222.593982:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829222.593984:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5859 00000100:00080000:0.0:1787829222.593986:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829222.593989:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829222.593992:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829222.594000:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790658560/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829222.594011:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829222.594014:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829222.594015:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829222.594020:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790658304:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829222.594021:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5859 00000100:00080000:0.0:1787829222.594027:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829222.594031:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829222.594035:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829222.594043:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790658688/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829222.594049:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829222.594052:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829222.594053:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829222.594056:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5859 00000100:00080000:0.0:1787829222.594059:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829222.594061:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829222.594064:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829222.594072:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790658816/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829222.594078:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829222.594082:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:3.0:1787829222.594083:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829222.594084:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829222.594089:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5859 00000100:00080000:0.0:1787829222.594093:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829222.594096:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829222.594100:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829222.594120:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790658944/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829222.594128:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829222.594130:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829222.594131:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829222.594137:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5859 00000100:00100000:3.0:1787829222.594138:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829222/real 1787829222] req@ffff9e6803e9bc00 x1874668790658304/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829222.594140:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829222.594147:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829222.594156:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829222.594162:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790658304/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829222.594168:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790659072/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829222.594178:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829222.594179:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829222.594185:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829222.594185:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829222.594188:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829222.594192:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790658304:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829222.594195:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5859 00000100:00080000:0.0:1787829222.594201:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829222.594204:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790658432:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829222.594205:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829222.594213:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829222.594224:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790659200/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829222.594232:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829222.594240:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829222.594245:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787829222.594245:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829222/real 1787829222] req@ffff9e6803e992c0 x1874668790658432/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829222.594254:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829222.594258:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790658432/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829222.594261:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5859 00000100:00080000:0.0:1787829222.594266:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829222.594273:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:3.0:1787829222.594274:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829222.594281:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:0.0:1787829222.594282:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829222.594288:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790658432:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829222.594295:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790659328/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829222.594321:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790658560:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829222.594334:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829222.594341:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829222/real 1787829222] req@ffff9e6803e98f00 x1874668790658560/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829222.594349:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790658560/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829222.594354:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829222.594356:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829222.594358:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790658560:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829222.594361:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790658688:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829222.594371:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829222.594377:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829222/real 1787829222] req@ffff9e6803e9ad00 x1874668790658688/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829222.594384:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790658688/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829222.594389:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829222.594390:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829222.594392:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790658688:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829222.594395:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790658816:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829222.594413:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829222.594420:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829222/real 1787829222] req@ffff9e6803e9a940 x1874668790658816/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829222.594427:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790658816/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829222.594433:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829222.594435:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829222.594436:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790658816:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829222.594440:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790658944:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829222.594449:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829222.594455:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829222/real 1787829222] req@ffff9e6803e98b40 x1874668790658944/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829222.594462:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790658944/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829222.594468:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829222.594470:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829222.594471:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790658944:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829222.594475:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790659072:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829222.594484:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829222.594489:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829222/real 1787829222] req@ffff9e6803e98780 x1874668790659072/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829222.594496:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790659072/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829222.594501:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829222.594503:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829222.594506:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790659072:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829222.594509:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790659200:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829222.594518:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829222.594523:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829222/real 1787829222] req@ffff9e6803e9a1c0 x1874668790659200/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829222.594529:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790659200/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829222.594534:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829222.594536:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829222.594538:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790659200:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829222.594541:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790659328:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829222.594550:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829222.594555:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829222/real 1787829222] req@ffff9e6803e9b840 x1874668790659328/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829222.594562:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790659328/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829222.594566:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829222.594568:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829222.594571:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790659328:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829223.105846:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829223.298097:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829224.129794:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829224.321829:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829225.153835:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829225.345929:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829226.177846:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829226.369885:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829227.201841:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829227.393918:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829227.713867:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829227.713881:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829227.713888:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829227.713901:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5864 00000100:00080000:3.0:1787829227.713917:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829227.714016:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829227.714056:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790659456/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829227.714094:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829227.714098:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829227.714101:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829227.714105:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5864 00000100:00080000:3.0:1787829227.714113:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829227.714118:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829227.714127:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829227.714131:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790659456:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829227.714140:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790659584/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829227.714150:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829227.714153:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829227.714154:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829227.714157:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5864 00000100:00080000:3.0:1787829227.714160:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829227.714162:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829227.714165:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829227.714181:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790659712/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829227.714189:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829227.714192:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829227.714194:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829227.714197:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829227.714201:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5864 00000100:00080000:3.0:1787829227.714204:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829227.714208:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829227.714212:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829227.714221:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790659840/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829227.714228:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829227.714232:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787829227.714233:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829227/real 1787829227] req@ffff9e680a2d7480 x1874668790659456/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829227.714234:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829227.714241:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5864 00000100:00080000:3.0:1787829227.714247:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829227.714252:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829227.714256:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829227.714260:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790659456/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829227.714268:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790659968/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829227.714271:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829227.714278:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829227.714285:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790659456:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829227.714292:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829227.714297:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829227.714299:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829227.714305:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5864 00000100:00080000:3.0:1787829227.714309:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829227.714310:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790659584:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829227.714318:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829227.714325:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829227.714338:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790660096/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829227.714342:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829227.714350:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829227.714353:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829227.714355:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829227.714359:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5864 00000100:00100000:0.0:1787829227.714362:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829227/real 1787829227] req@ffff9e680a2d5e00 x1874668790659584/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829227.714364:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829227.714372:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829227.714380:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829227.714384:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790659584/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829227.714391:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790660224/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829227.714394:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829227.714401:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829227.714408:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790659584:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829227.714419:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790659712:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829227.714422:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829227.714426:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829227.714428:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829227.714432:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5864 00000100:00080000:3.0:1787829227.714435:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829227.714440:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787829227.714441:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787829227.714447:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829227.714455:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829227/real 1787829227] req@ffff9e680a2d6d00 x1874668790659712/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829227.714459:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790660352/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829227.714473:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829227.714474:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790659712/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829227.714480:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829227.714484:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829227.714491:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829227.714491:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5864 00000100:00080000:0.0:1787829227.714499:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829227.714502:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790659712:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829227.714506:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790659840:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829227.714506:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829227.714514:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829227.714519:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829227.714528:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829227.714533:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790660480/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829227.714538:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829227/real 1787829227] req@ffff9e680a2d61c0 x1874668790659840/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829227.714553:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790659840/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829227.714564:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829227.714566:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829227.714569:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790659840:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829227.714573:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790659968:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829227.714583:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829227.714589:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829227/real 1787829227] req@ffff9e680a2d4f00 x1874668790659968/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829227.714596:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790659968/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829227.714601:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829227.714603:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829227.714605:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790659968:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829227.714616:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790660096:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829227.714627:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829227.714633:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829227/real 1787829227] req@ffff9e680a2d7480 x1874668790660096/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829227.714640:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790660096/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829227.714646:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829227.714647:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829227.714650:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790660096:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829227.714653:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790660224:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829227.714663:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829227.714668:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829227/real 1787829227] req@ffff9e680a2d5a40 x1874668790660224/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829227.714675:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790660224/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829227.714680:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829227.714681:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829227.714683:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790660224:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829227.714686:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790660352:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829227.714714:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829227.714721:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829227/real 1787829227] req@ffff9e680a2d4000 x1874668790660352/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829227.714728:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790660352/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829227.714733:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829227.714735:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829227.714737:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790660352:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829227.714740:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790660480:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829227.714751:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829227.714758:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829227/real 1787829227] req@ffff9e680a2d4780 x1874668790660480/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829227.714765:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790660480/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829227.714769:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829227.714771:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829227.714773:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790660480:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829228.225817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829228.417856:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829229.249810:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829229.441895:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829230.273765:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829230.465824:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829231.297781:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829231.489884:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829232.321791:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829232.513883:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829232.833847:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829232.833858:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829232.833862:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829232.833872:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5869 00000100:00080000:3.0:1787829232.833884:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829232.833891:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829232.833925:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790660608/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829232.833952:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829232.833956:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829232.833958:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829232.833963:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5869 00000100:00080000:3.0:1787829232.833968:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829232.833974:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829232.833984:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829232.833985:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790660608:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829232.834003:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790660736/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829232.834019:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829232.834022:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829232.834024:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829232.834028:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5869 00000100:00080000:3.0:1787829232.834034:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829232.834056:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829232.834063:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829232.834073:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790660864/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829232.834111:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790660736:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829232.834146:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790660864:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829232.834157:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829232.834162:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829232.834164:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829232.834168:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5869 00000100:00080000:3.0:1787829232.834295:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829232.834302:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829232.834310:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829232.834324:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790660992/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829232.834341:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829232.834344:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829232.834347:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829232.834349:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5869 00000100:00080000:3.0:1787829232.834353:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829232.834358:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829232.834364:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829232.834386:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790661120/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829232.834406:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829232.834410:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829232.834412:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829232.834418:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5869 00000100:00080000:3.0:1787829232.834422:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829232.834429:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829232.834430:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790660992:192.168.203.109@tcp:8:kworker.0 00000008:00100000:3.0:1787829232.834435:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829232.834449:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790661248/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829232.834461:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829232.834466:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829232.834468:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829232.834472:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5869 00000100:00100000:0.0:1787829232.834477:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790661120:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829232.834479:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829232.834488:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829232.834494:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829232.834510:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790661376/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829232.834518:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790661248:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829232.834524:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829232.834529:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829232.834531:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829232.834537:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5869 00000100:00080000:3.0:1787829232.834543:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829232.834548:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829232.834553:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790661376:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787829232.834556:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829232.834574:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790661504/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829232.834587:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829232.834591:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787829232.834595:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790661504:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829232.834596:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829232.834605:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5869 00000100:00080000:3.0:1787829232.834610:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829232.834615:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829232.834621:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829232.834635:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790661632/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829232.834673:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790661632:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829233.345843:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829233.537967:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829234.369761:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829234.561848:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829235.393815:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829235.585880:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787829235.905978:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:1.0:1787829235.906034:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787829235.906052:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829235.906108:0:2386:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:1.0:1787829235.906124:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829235.906133:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829235.906143:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1787829235.906150:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829232/real 1787829235] req@ffff9e680a2d5a40 x1874668790661632/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787829235.906154:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829235.906165:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829235.906185:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1787829235.906195:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790661632/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787829235.906199:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00080000:0.0:1787829235.906216:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829235.906223:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829235.906228:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790661632:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829235.906236:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829232/real 1787829235] req@ffff9e680a2d4000 x1874668790661504/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829235.906250:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790661504/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829235.906258:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829235.906267:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829235.906271:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790661504:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829235.906278:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829232/real 1787829235] req@ffff9e680a2d4780 x1874668790661376/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829235.906291:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790661376/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829235.906299:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829235.906302:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829235.906305:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790661376:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829235.906312:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829232/real 1787829235] req@ffff9e680a2d43c0 x1874668790661248/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829235.906327:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790661248/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829235.906337:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829235.906341:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829235.906345:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790661248:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829235.906352:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829232/real 1787829235] req@ffff9e680a2d70c0 x1874668790660992/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829235.906361:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790660992/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829235.906370:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829235.906373:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829235.906377:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790660992:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829235.906383:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829232/real 1787829235] req@ffff9e680a2d6580 x1874668790661120/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829235.906393:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790661120/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829235.906401:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829235.906404:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829235.906407:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790661120:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829235.906415:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829232/real 1787829235] req@ffff9e680a2d4b40 x1874668790660736/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829235.906427:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790660736/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829235.906433:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829235.906436:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829235.906438:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790660736:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829235.906442:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829232/real 1787829235] req@ffff9e680a2d7c00 x1874668790660864/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829235.906449:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790660864/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829235.906454:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829235.906457:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829235.906459:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790660864:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829235.906464:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829232/real 1787829235] req@ffff9e680a2d52c0 x1874668790660608/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829235.906475:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790660608/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829235.906482:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829235.906485:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829235.906488:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790660608:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829236.417800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829236.609883:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829237.441837:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829237.633878:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829237.953811:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829237.953826:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829237.953833:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829237.953848:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5874 00000100:00080000:3.0:1787829237.953861:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829237.953869:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829237.953913:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790661760/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829237.953936:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829237.953940:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829237.953942:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829237.953946:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5874 00000100:00080000:3.0:1787829237.953949:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829237.953951:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829237.953961:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829237.953971:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790661888/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829237.953974:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790661760:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829237.953983:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829237.953993:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829237.953994:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829237.953998:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5874 00000100:00080000:3.0:1787829237.954000:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829237.954017:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829237.954022:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829237.954035:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790662016/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829237.954052:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829237.954057:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829237.954059:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829237.954064:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5874 00000100:00080000:3.0:1787829237.954070:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829237.954071:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829237.954084:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829237.954093:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829237.954112:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790662144/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829237.954126:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829237/real 1787829237] req@ffff9e680a2d7480 x1874668790661760/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829237.954126:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829237.954150:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787829237.954152:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790661760/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829237.954154:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829237.954172:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5874 00000100:00080000:3.0:1787829237.954185:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829237.954190:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829237.954197:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787829237.954203:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:3.0:1787829237.954211:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829237.954213:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790661760:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829237.954237:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790662272/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829237.954254:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829237.954255:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790661888:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829237.954269:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829237.954273:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829237.954281:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5874 00000100:00080000:3.0:1787829237.954289:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829237.954297:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787829237.954307:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787829237.954312:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829237.954329:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829237/real 1787829237] req@ffff9e680a2d4f00 x1874668790661888/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829237.954337:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790662400/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829237.954368:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829237.954372:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790661888/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829237.954375:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829237.954381:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829237.954396:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5874 00000100:00080000:3.0:1787829237.954402:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829237.954411:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787829237.954412:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787829237.954419:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829237.954424:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829237.954431:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790661888:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829237.954439:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790662528/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829237.954445:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790662016:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829237.954473:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829237.954478:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829237.954481:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829237.954489:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5874 00000800:00000100:0.0:1787829237.954494:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829237.954509:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829237.954510:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829237/real 1787829237] req@ffff9e680a2d61c0 x1874668790662016/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829237.954525:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829237.954540:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829237.954546:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790662016/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829237.954562:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790662656/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829237.954579:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829237.954593:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829237.954607:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790662016:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829237.954632:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790662144:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829237.956026:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829237.956050:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829237/real 1787829237] req@ffff9e680a2d6d00 x1874668790662144/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829237.956066:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790662144/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829237.956077:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829237.956082:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829237.956085:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790662144:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829237.956113:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790662272:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829237.956135:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829237.956145:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829237.956151:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787829237.956152:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829237/real 1787829237] req@ffff9e680a2d5e00 x1874668790662272/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829237.956154:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829237.956165:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5874 00000100:00080000:3.0:1787829237.956172:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829237.956173:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790662272/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829237.956181:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787829237.956190:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787829237.956195:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829237.956196:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829237.956203:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790662272:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829237.956212:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790662400:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829237.956221:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790662784/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829237.956235:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829237.956248:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829237/real 1787829237] req@ffff9e680a2d7480 x1874668790662400/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829237.956260:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790662400/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829237.956271:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829237.956274:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829237.956277:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790662400:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829237.956284:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790662528:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829237.956300:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829237.956309:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829237/real 1787829237] req@ffff9e680a2d52c0 x1874668790662528/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829237.956320:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790662528/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829237.956329:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829237.956332:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829237.956335:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790662528:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829237.956351:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790662656:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829237.956372:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829237.956384:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829237/real 1787829237] req@ffff9e680a2d7c00 x1874668790662656/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829237.956396:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790662656/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829237.956411:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829237.956418:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829237.956421:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790662656:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829237.956428:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790662784:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829237.956450:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829237.956460:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829237/real 1787829237] req@ffff9e680a2d4b40 x1874668790662784/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829237.956472:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790662784/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829262 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829237.956484:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829237.956488:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829237.956491:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790662784:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829238.465840:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829238.657917:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829239.489807:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829239.681880:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829240.513855:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829240.705840:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829241.537836:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829241.729892:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829242.561848:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829242.753881:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829243.073814:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829243.073826:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829243.073831:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829243.073841:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5879 00000100:00080000:1.0:1787829243.073851:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829243.073857:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829243.073894:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790662912/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829243.073925:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829243.073929:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829243.073931:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829243.073937:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5879 00000100:00080000:1.0:1787829243.073944:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829243.073950:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829243.073962:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829243.073981:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790662912:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787829243.073994:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790663040/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829243.074009:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829243.074014:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829243.074016:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829243.074021:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5879 00000100:00080000:1.0:1787829243.074025:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829243.074030:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829243.074036:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829243.074046:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790663168/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829243.074054:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829243.074057:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829243.074058:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829243.074062:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5879 00000100:00080000:1.0:1787829243.074065:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829243.074073:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829243.074079:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829243.074085:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829243.074095:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790663296/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829243.074103:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829243.074106:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829243.074107:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829243.074110:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5879 00000100:00080000:1.0:1787829243.074112:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829243.074114:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829243.074120:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829243.074131:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829243/real 1787829243] req@ffff9e6831aaad00 x1874668790662912/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829243.074135:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790663424/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829243.074193:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829243.074197:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829243.074200:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829243.074205:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790662912/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829243.074211:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5879 00000100:00080000:1.0:1787829243.074219:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829243.074232:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829243.074241:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829243.074242:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829243.074255:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829243.074260:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790663552/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829243.074262:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790662912:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829243.074283:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829243.074286:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829243.074287:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829243.074292:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5879 00000100:00080000:1.0:1787829243.074294:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829243.074298:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829243.074305:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829243.074307:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790663040:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829243.074317:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790663680/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829243.074328:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829243.074330:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829243.074331:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829243.074334:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5879 00000100:00080000:1.0:1787829243.074337:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829243.074343:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829243.074348:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829243.074355:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829243.074368:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829243/real 1787829243] req@ffff9e6831aab480 x1874668790663040/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829243.074375:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790663808/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829243.074408:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790663040/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829243.074461:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829243.074467:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829243.074468:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829243.074473:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790663040:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829243.074485:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787829243.074490:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790663168:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829243.074492:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829243.074502:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5879 00000100:00080000:1.0:1787829243.074522:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829243.074527:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787829243.074532:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1787829243.074534:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829243.074544:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829243/real 1787829243] req@ffff9e6831aa8780 x1874668790663168/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829243.074550:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790663936/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829243.074578:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790663168/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829243.074588:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829243.074592:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829243.074595:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790663168:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829243.074619:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790663296:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829243.074634:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829243.074641:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829243/real 1787829243] req@ffff9e6831aaa1c0 x1874668790663296/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829243.074648:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790663296/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829243.074654:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829243.074656:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829243.074657:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790663296:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829243.074661:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790663424:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829243.074671:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829243.074677:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829243/real 1787829243] req@ffff9e6831aa8000 x1874668790663424/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829243.074684:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790663424/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829243.074689:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829243.074691:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829243.074693:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790663424:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829243.074696:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790663552:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829243.074706:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829243.074712:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829243/real 1787829243] req@ffff9e6831aa9a40 x1874668790663552/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829243.074720:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790663552/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829243.074745:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829243.074748:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829243.074750:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790663552:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829243.074765:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790663680:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829243.074778:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829243.074785:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829243/real 1787829243] req@ffff9e6831aabc00 x1874668790663680/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829243.074793:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790663680/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829243.074799:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829243.074801:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829243.074803:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790663680:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829243.074807:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790663808:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829243.074817:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829243.074823:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829243/real 1787829243] req@ffff9e6831aaa940 x1874668790663808/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829243.074830:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790663808/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829243.074840:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829243.074842:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829243.074844:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790663808:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829243.074848:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790663936:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829243.074858:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829243.074863:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829243/real 1787829243] req@ffff9e6831aa8b40 x1874668790663936/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829243.074869:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790663936/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829243.074874:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829243.074876:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829243.074878:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790663936:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829243.585864:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829243.777917:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829244.609799:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829244.801833:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829245.633873:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829245.825880:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829246.657863:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829246.850011:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829247.681848:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829247.873964:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829248.193785:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829248.193798:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829248.193804:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829248.193818:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5884 00000100:00080000:1.0:1787829248.193834:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829248.193861:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829248.193924:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790664064/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829248.193996:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829248.194002:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829248.194005:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829248.194013:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5884 00000100:00080000:1.0:1787829248.194020:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829248.194028:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829248.194039:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790664064:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787829248.194041:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829248.194063:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790664192/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829248.194081:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829248.194086:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829248.194089:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829248.194095:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5884 00000100:00080000:1.0:1787829248.194140:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829248.194148:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829248.194155:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829248.194157:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829248.194175:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790664320/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829248.194186:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829248.194190:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829248.194192:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829248.194196:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5884 00000100:00080000:1.0:1787829248.194201:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829248.194207:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829248.194214:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829248.194230:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790664448/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829248.194255:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829248.194260:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829248.194263:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829248.194269:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5884 00000100:00080000:1.0:1787829248.194274:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829248.194276:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829248/real 1787829248] req@ffff9e6831aa9680 x1874668790664064/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829248.194282:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829248.194295:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829248.194309:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790664064/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829248.194321:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790664576/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829248.194326:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829248.194335:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829248.194345:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790664064:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829248.194352:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829248.194358:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829248.194360:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829248.194366:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5884 00000100:00080000:1.0:1787829248.194371:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829248.194375:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829248.194382:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829248.194383:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790664192:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829248.194398:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790664704/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829248.194414:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829248.194418:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829248.194420:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829248.194423:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5884 00000100:00080000:1.0:1787829248.194430:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829248.194437:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829248.194438:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829248.194447:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829248.194486:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790664832/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829248.194487:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829248/real 1787829248] req@ffff9e6831aaa580 x1874668790664192/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829248.194513:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829248.194520:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829248.194522:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829248.194527:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5884 00000100:00080000:1.0:1787829248.194531:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829248.194533:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790664192/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829248.194566:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829248.194568:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829248.194578:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829248.194580:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829248.194589:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790664192:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829248.194594:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790664960/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829248.194607:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790664320:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829248.194609:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829248.194618:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829248.194621:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829248.194626:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5884 00000100:00080000:1.0:1787829248.194630:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829248.194633:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829248.194637:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829248.194649:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790665088/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829248.194660:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829248.194689:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829248/real 1787829248] req@ffff9e6831aa8f00 x1874668790664320/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829248.194711:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790664320/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829248.194724:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829248.194775:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829248.194781:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790664320:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829248.194791:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790664448:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829248.194841:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829248.194860:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829248/real 1787829248] req@ffff9e6831aa9e00 x1874668790664448/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829248.194880:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790664448/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829248.194910:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829248.194916:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829248.194921:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790664448:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829248.194958:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790664576:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829248.195008:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829248.195025:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829248/real 1787829248] req@ffff9e6831aa8b40 x1874668790664576/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829248.195043:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790664576/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829248.195055:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829248.195061:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829248.195066:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790664576:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829248.195113:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790664704:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829248.195148:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829248.195167:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829248/real 1787829248] req@ffff9e6831aaa940 x1874668790664704/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829248.195180:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790664704/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829248.195205:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829248.195211:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829248.195216:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790664704:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829248.195226:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790664832:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829248.195260:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829248.195278:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829248/real 1787829248] req@ffff9e6831aabc00 x1874668790664832/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829248.195292:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790664832/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829248.195302:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829248.195305:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829248.195308:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790664832:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829248.195315:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790664960:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829248.195336:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829248.195346:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829248/real 1787829248] req@ffff9e6831aa9a40 x1874668790664960/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829248.195358:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790664960/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829248.195366:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829248.195368:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829248.195372:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790664960:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829248.195378:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790665088:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829248.195394:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829248.195402:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829248/real 1787829248] req@ffff9e6831aa8000 x1874668790665088/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829248.195413:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790665088/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829248.195421:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829248.195424:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829248.195426:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790665088:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829248.705855:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:0.0:1787829248.898047:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829249.729838:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829249.921990:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829250.753816:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829250.945888:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829251.777817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829251.969919:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829252.801834:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:0.0:1787829252.993911:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829253.313834:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829253.313846:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829253.313851:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829253.313862:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5890 00000100:00080000:3.0:1787829253.313875:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829253.313882:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829253.313940:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790665216/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829253.313965:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829253.313968:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829253.313970:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829253.313973:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5890 00000100:00080000:3.0:1787829253.313976:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829253.313980:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829253.313991:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829253.314003:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790665216:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829253.314004:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790665344/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829253.314018:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829253.314021:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829253.314023:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829253.314026:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5890 00000100:00080000:3.0:1787829253.314028:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829253.314030:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829253.314034:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829253.314042:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790665472/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829253.314049:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829253.314052:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829253.314053:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829253.314056:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5890 00000100:00080000:3.0:1787829253.314059:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829253.314061:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829253.314064:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829253.314072:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790665600/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829253.314079:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829253.314084:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829253.314085:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829253.314088:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829253.314093:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5890 00000100:00080000:3.0:1787829253.314095:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829253.314098:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829253.314102:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829253.314111:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790665728/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829253.314116:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829253.314119:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829253.314120:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829253.314124:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5890 00000100:00080000:3.0:1787829253.314128:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829253.314133:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829253/real 1787829253] req@ffff9e680a2d6580 x1874668790665216/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829253.314134:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829253.314141:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829253.314155:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790665856/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829253.314161:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790665216/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829253.314167:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829253.314175:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829253.314178:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829253.314185:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829253.314185:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5890 00000100:00080000:3.0:1787829253.314187:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829253.314191:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829253.314193:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829253.314199:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790665216:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787829253.314199:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829253.314212:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790665984/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829253.314219:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829253.314221:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829253.314222:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829253.314227:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5890 00000100:00080000:3.0:1787829253.314229:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829253.314232:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790665344:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829253.314235:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829253.314242:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829253.314252:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790666112/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829253.314262:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829253.314262:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829253.314268:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829253.314270:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829253.314275:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5890 00000100:00100000:0.0:1787829253.314279:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829253/real 1787829253] req@ffff9e680a2d70c0 x1874668790665344/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829253.314280:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829253.314287:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829253.314296:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790665344/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829253.314307:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829253.314308:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829253.314314:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829253.314317:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790665344:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829253.314322:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790665472:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829253.314329:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790666240/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829253.314339:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829253.314349:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829253/real 1787829253] req@ffff9e680a2d43c0 x1874668790665472/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829253.314358:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790665472/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829253.314364:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829253.314367:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829253.314369:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790665472:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829253.314373:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790665600:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829253.314385:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829253.314391:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829253/real 1787829253] req@ffff9e680a2d4780 x1874668790665600/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829253.314403:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790665600/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829253.314409:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829253.314410:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829253.314412:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790665600:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829253.314415:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790665728:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829253.314425:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829253.314430:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829253/real 1787829253] req@ffff9e680a2d4000 x1874668790665728/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829253.314437:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790665728/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829253.314443:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829253.314445:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829253.314447:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790665728:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829253.314450:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790665856:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829253.314460:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829253.314466:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829253/real 1787829253] req@ffff9e680a2d5a40 x1874668790665856/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829253.314473:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790665856/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829253.314487:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829253.314491:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829253.314494:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790665856:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829253.314501:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790665984:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829253.314522:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829253.314534:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829253/real 1787829253] req@ffff9e680a2d4b40 x1874668790665984/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829253.314548:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790665984/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829253.314556:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829253.314560:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829253.314564:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790665984:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829253.314588:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790666112:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829253.314611:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829253.314624:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829253/real 1787829253] req@ffff9e680a2d7c00 x1874668790666112/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829253.314637:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790666112/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829253.314646:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829253.314649:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829253.314653:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790666112:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829253.314660:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790666240:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829253.314682:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829253.314716:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829253/real 1787829253] req@ffff9e680a2d52c0 x1874668790666240/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829253.314731:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790666240/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829253.314741:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829253.314744:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829253.314748:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790666240:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829253.825782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829254.017907:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829254.849839:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829255.041913:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829255.873840:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829256.066211:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829256.897864:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829257.089857:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829257.921817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829258.113819:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829258.433802:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829258.433810:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829258.433813:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829258.433821:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5895 00000100:00080000:1.0:1787829258.433831:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829258.433837:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829258.433884:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790666368/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829258.433920:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829258.433923:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829258.433924:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829258.433928:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5895 00000100:00080000:1.0:1787829258.433931:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829258.433935:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829258.433943:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829258.433954:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790666496/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829258.433965:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829258.433970:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790666368:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829258.433970:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829258.433972:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829258.433988:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5895 00000100:00080000:1.0:1787829258.433992:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829258.433996:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829258.434004:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829258.434033:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790666624/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829258.434044:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829258.434048:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829258.434050:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829258.434056:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5895 00000100:00080000:1.0:1787829258.434104:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829258.434108:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829258.434114:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829258.434117:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829258.434129:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790666752/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829258.434139:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829258.434152:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829258.434154:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829258.434159:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5895 00000100:00080000:1.0:1787829258.434165:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829258.434168:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829258.434171:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829258.434181:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790666880/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829258.434189:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829258.434194:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787829258.434195:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829258/real 1787829258] req@ffff9e6831aaa1c0 x1874668790666368/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829258.434200:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829258.434219:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5895 00000100:00080000:1.0:1787829258.434221:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829258.434224:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829258.434227:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829258.434249:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790667008/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829258.434257:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790666368/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829258.434267:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829258.434273:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829258.434274:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829258.434281:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829258.434283:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829258.434287:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790666368:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829258.434300:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5895 00000100:00080000:1.0:1787829258.434306:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829258.434310:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829258.434316:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790666496:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787829258.434316:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829258.434337:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790667136/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829258.434348:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829258.434352:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829258.434354:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829258.434365:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829258.434366:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829258/real 1787829258] req@ffff9e6831aa8780 x1874668790666496/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829258.434377:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5895 00000100:00080000:1.0:1787829258.434386:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829258.434392:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790666496/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829258.434393:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829258.434406:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829258.434411:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829258.434415:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829258.434418:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790666496:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829258.434426:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790666624:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829258.434427:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790667264/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829258.434444:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829258.434448:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829258.434454:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829258.434458:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829258.434460:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829258/real 1787829258] req@ffff9e6831aab480 x1874668790666624/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829258.434468:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5895 00000100:00080000:1.0:1787829258.434473:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829258.434478:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829258.434485:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829258.434493:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790666624/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829258.434499:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790667392/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829258.434525:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829258.434539:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829258.434545:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790666624:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829258.434551:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790666752:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829258.434568:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829258.434575:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829258/real 1787829258] req@ffff9e6831aaad00 x1874668790666752/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829258.434583:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790666752/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829258.434589:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829258.434591:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829258.434594:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790666752:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829258.434597:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790666880:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829258.434607:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829258.434613:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829258/real 1787829258] req@ffff9e6831aa8000 x1874668790666880/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829258.434621:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790666880/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829258.434627:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829258.434629:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829258.434631:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790666880:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829258.434634:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790667008:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829258.434644:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829258.434651:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829258/real 1787829258] req@ffff9e6831aa9a40 x1874668790667008/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829258.434662:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790667008/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829258.434673:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829258.434676:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829258.434681:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790667008:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829258.434727:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790667136:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829258.434746:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829258.434754:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829258/real 1787829258] req@ffff9e6831aabc00 x1874668790667136/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829258.434763:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790667136/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829258.434769:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829258.434771:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829258.434773:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790667136:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829258.434777:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790667264:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829258.434788:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829258.434794:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829258/real 1787829258] req@ffff9e6831aaa940 x1874668790667264/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829258.434802:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790667264/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829258.434807:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829258.434809:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829258.434811:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790667264:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829258.434814:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790667392:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829258.434824:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829258.434831:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829258/real 1787829258] req@ffff9e6831aa8b40 x1874668790667392/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829258.434839:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790667392/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829258.434844:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829258.434846:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829258.434848:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790667392:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829258.945918:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829259.137884:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829259.969904:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829260.162557:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829260.993807:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829261.185926:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829262.017943:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829262.210041:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829263.041761:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829263.233886:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829263.553811:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829263.553821:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829263.553838:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829263.553851:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5900 00000100:00080000:3.0:1787829263.553872:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829263.553881:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829263.553927:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790667520/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829263.553950:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829263.553953:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829263.553954:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829263.553957:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5900 00000100:00080000:3.0:1787829263.553960:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829263.553965:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829263.553977:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829263.553991:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790667520:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829263.553994:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790667648/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829263.554009:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829263.554013:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829263.554016:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829263.554021:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5900 00000100:00080000:3.0:1787829263.554025:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829263.554030:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829263.554036:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829263.554047:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790667776/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829263.554060:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829263.554063:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829263.554065:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829263.554068:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5900 00000100:00080000:3.0:1787829263.554071:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829263.554076:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787829263.554078:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787829263.554082:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829263.554090:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790667904/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829263.554098:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829263.554100:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829263.554102:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829263.554104:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5900 00000100:00080000:3.0:1787829263.554106:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829263.554108:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829263.554111:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829263.554119:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790668032/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829263.554125:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829263.554128:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829263/real 1787829263] req@ffff9e680a2d7480 x1874668790667520/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829263.554129:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829263.554132:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829263.554140:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5900 00000100:00080000:3.0:1787829263.554144:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829263.554149:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829263.554153:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829263.554158:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790667520/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829263.554163:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790668160/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829263.554181:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829263.554183:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829263.554185:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829263.554185:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829263.554193:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5900 00000100:00080000:3.0:1787829263.554196:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829263.554197:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829263.554201:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829263.554206:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829263.554234:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790668288/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829263.554239:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790667520:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829263.554243:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829263.554251:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829263.554253:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829263.554257:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5900 00000100:00080000:3.0:1787829263.554259:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829263.554263:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829263.554266:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829263.554273:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790668416/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829263.554301:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790667648:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829263.554329:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829263.554334:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829263.554336:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829263.554359:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5900 00000100:00080000:3.0:1787829263.554364:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829263.554367:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829263.554371:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829263.554378:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829263.554384:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790668544/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829263.554404:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829263/real 1787829263] req@ffff9e680a2d5e00 x1874668790667648/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829263.554447:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790667648/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829263.554460:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829263.554462:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829263.554465:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790667648:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829263.554470:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790667776:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829263.554487:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829263.554495:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829263/real 1787829263] req@ffff9e680a2d6d00 x1874668790667776/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829263.554503:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790667776/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829263.554509:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829263.554511:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829263.554512:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790667776:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829263.554524:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790667904:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829263.554537:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829263.554543:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829263/real 1787829263] req@ffff9e680a2d61c0 x1874668790667904/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829263.554551:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790667904/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829263.554556:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829263.554559:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829263.554561:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790667904:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829263.554565:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790668032:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829263.554576:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829263.554581:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829263/real 1787829263] req@ffff9e680a2d4f00 x1874668790668032/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829263.554589:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790668032/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829263.554594:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829263.554596:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829263.554598:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790668032:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829263.554602:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790668160:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829263.554612:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829263.554618:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829263/real 1787829263] req@ffff9e680a2d52c0 x1874668790668160/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829263.554626:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790668160/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829263.554632:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829263.554634:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829263.554636:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790668160:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829263.554639:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790668288:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829263.554649:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829263.554656:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829263/real 1787829263] req@ffff9e680a2d7c00 x1874668790668288/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829263.554663:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790668288/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829263.554669:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829263.554671:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829263.554672:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790668288:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829263.554689:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790668416:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829263.554719:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829263.554727:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829263/real 1787829263] req@ffff9e680a2d4b40 x1874668790668416/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829263.554734:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790668416/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829263.554739:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829263.554742:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829263.554744:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790668416:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829263.554748:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790668544:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829263.554758:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829263.554764:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829263/real 1787829263] req@ffff9e680a2d5a40 x1874668790668544/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829263.554771:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790668544/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829263.554777:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829263.554779:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829263.554781:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790668544:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829264.065883:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829264.257947:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829265.089774:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829265.281829:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829266.114015:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829266.305921:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829267.137788:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829267.329829:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829268.161819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829268.353984:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829268.673911:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829268.673926:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829268.673932:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829268.673947:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5905 00000100:00080000:0.0:1787829268.673963:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829268.673974:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829268.674024:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790668672/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829268.674062:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829268.674067:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829268.674071:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829268.674076:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5905 00000100:00080000:0.0:1787829268.674081:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829268.674088:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829268.674102:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829268.674104:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790668672:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787829268.674127:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790668800/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829268.674141:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829268.674145:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829268.674149:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829268.674154:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5905 00000100:00080000:0.0:1787829268.674159:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829268.674165:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829268.674171:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829268.674185:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790668928/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829268.674197:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829268.674202:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829268.674205:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829268.674211:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5905 00000100:00080000:0.0:1787829268.674216:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829268.674223:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829268.674231:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829268.674246:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790669056/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829268.674258:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829268.674261:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829268.674263:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829268.674269:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5905 00000100:00080000:0.0:1787829268.674273:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829268.674279:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829268.674285:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787829268.674297:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829268.674303:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790669184/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829268.674316:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829268.674320:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829268.674322:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829268.674326:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5905 00000100:00080000:0.0:1787829268.674330:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829268.674335:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829268.674341:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829268.674354:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790669312/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829268.674364:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829268.674368:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829268.674370:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829268.674375:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5905 00000100:00080000:0.0:1787829268.674379:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829268.674384:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829268.674389:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829268.674400:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790669440/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829268.674410:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829268.674414:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829268.674416:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829268.674421:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5905 00000100:00080000:0.0:1787829268.674425:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829268.674430:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829268.674435:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829268.674448:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790669568/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829268.674459:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829268.674463:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829268.674465:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829268.674471:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5905 00000100:00080000:0.0:1787829268.674476:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829268.674481:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829268.674486:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829268.674499:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790669696/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829268.674556:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829268/real 1787829268] req@ffff9e6803e9b480 x1874668790668672/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829268.674582:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790668672/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829268.674596:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829268.674602:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829268.674618:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790668672:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829268.674648:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790668800:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829268.674676:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829268.674693:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829268/real 1787829268] req@ffff9e6803e99e00 x1874668790668800/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829268.674742:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790668800/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829268.674754:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829268.674758:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829268.674764:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790668800:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829268.674773:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790668928:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829268.674826:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829268.674839:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829268/real 1787829268] req@ffff9e6803e99680 x1874668790668928/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829268.674852:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790668928/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829268.674861:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829268.674864:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829268.674868:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790668928:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829268.674874:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790669056:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829268.674895:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829268.674906:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829268/real 1787829268] req@ffff9e6803e9a580 x1874668790669056/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829268.674919:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790669056/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829268.674928:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829268.674932:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829268.674935:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790669056:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829268.674941:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790669184:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829268.674961:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829268.674973:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829268/real 1787829268] req@ffff9e6803e9b840 x1874668790669184/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829268.674984:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790669184/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829268.674993:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829268.674996:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829268.675000:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790669184:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829268.675006:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790669312:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829268.675025:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829268.675037:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829268/real 1787829268] req@ffff9e6803e9a1c0 x1874668790669312/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829268.675050:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790669312/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829268.675060:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829268.675064:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829268.675068:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790669312:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829268.675074:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790669440:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829268.675094:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829268.675106:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829268/real 1787829268] req@ffff9e6803e98780 x1874668790669440/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829268.675118:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790669440/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829268.675126:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829268.675130:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829268.675133:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790669440:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829268.675140:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790669568:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829268.675159:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829268.675170:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829268/real 1787829268] req@ffff9e6803e98b40 x1874668790669568/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829268.675183:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790669568/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829268.675191:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829268.675195:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829268.675199:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790669568:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829268.675205:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790669696:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829268.675225:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829268.675236:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829268/real 1787829268] req@ffff9e6803e9a940 x1874668790669696/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829268.675247:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790669696/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829268.675255:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829268.675258:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829268.675262:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790669696:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829269.185817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829269.377884:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829270.209858:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829270.401941:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829271.233793:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829271.425909:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829272.257825:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829272.449842:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829273.281794:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829273.473932:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829273.793834:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829273.793846:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829273.793850:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829273.793861:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5910 00000100:00080000:0.0:1787829273.793872:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829273.793880:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829273.793935:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790669824/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829273.793962:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829273.793966:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829273.793967:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829273.793970:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5910 00000100:00080000:0.0:1787829273.793973:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829273.793975:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829273.793985:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829273.793994:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790669952/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829273.794002:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829273.794005:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829273.794007:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829273.794009:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5910 00000100:00080000:0.0:1787829273.794013:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829273.794017:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829273.794021:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829273.794035:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790670080/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829273.794035:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790669824:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829273.794056:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829273.794063:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829273.794066:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829273.794070:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5910 00000100:00080000:0.0:1787829273.794075:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829273.794080:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829273.794086:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829273.794101:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790670208/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829273.794108:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829273.794111:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829273.794114:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829273.794117:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5910 00000100:00080000:0.0:1787829273.794121:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829273.794124:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829273.794128:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787829273.794135:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829273.794145:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790670336/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829273.794158:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829273.794161:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829273.794164:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829273.794169:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5910 00000100:00080000:0.0:1787829273.794173:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829273.794180:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829273.794187:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829273.794196:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829273/real 1787829273] req@ffff9e6803e9ad00 x1874668790669824/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829273.794205:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790670464/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829273.794229:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829273.794232:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790669824/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829273.794235:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829273.794241:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829273.794257:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5910 00000100:00080000:3.0:1787829273.794270:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829273.794275:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829273.794283:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:3.0:1787829273.794288:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:0.0:1787829273.794295:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829273.794299:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790669824:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829273.794312:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790669952:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829273.794325:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790670592/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829273.794335:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829273.794338:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829273.794341:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829273.794345:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5910 00000100:00080000:0.0:1787829273.794348:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829273.794351:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829273.794356:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829273.794362:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829273.794378:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790670720/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829273.794380:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829273/real 1787829273] req@ffff9e6803e98f00 x1874668790669952/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829273.794422:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829273.794436:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829273.794442:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829273.794459:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5910 00000100:00100000:3.0:1787829273.794465:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790669952/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829273.794468:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829273.794480:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829273.794492:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787829273.794494:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829273.794504:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829273.794510:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790670848/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829273.794512:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790669952:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829273.794583:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790670080:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829273.794610:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829273.794624:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829273/real 1787829273] req@ffff9e6803e992c0 x1874668790670080/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829273.794639:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790670080/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829273.794647:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829273.794652:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829273.794655:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790670080:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829273.794663:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790670208:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829273.794684:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829273.794697:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829273/real 1787829273] req@ffff9e6803e9bc00 x1874668790670208/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829273.794733:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790670208/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829273.794751:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829273.794771:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829273.794775:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790670208:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829273.794783:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790670336:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829273.794806:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829273.794818:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829273/real 1787829273] req@ffff9e6803e9a940 x1874668790670336/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829273.794831:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790670336/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829273.794841:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829273.794845:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829273.794849:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790670336:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829273.794856:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790670464:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829273.794876:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829273.794888:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829273/real 1787829273] req@ffff9e6803e98b40 x1874668790670464/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829273.794901:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790670464/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829273.794928:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829273.794931:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829273.794936:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790670464:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829273.794942:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790670592:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829273.794964:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829273.794976:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829273/real 1787829273] req@ffff9e6803e98780 x1874668790670592/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829273.794990:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790670592/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829273.794998:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829273.795002:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829273.795005:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790670592:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829273.795013:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790670720:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829273.795035:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829273.795047:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829273/real 1787829273] req@ffff9e6803e9a1c0 x1874668790670720/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829273.795060:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790670720/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829273.795069:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829273.795073:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829273.795076:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790670720:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829273.795083:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790670848:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829273.795105:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829273.795117:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829273/real 1787829273] req@ffff9e6803e9b840 x1874668790670848/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829273.795129:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790670848/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829273.795138:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829273.795141:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829273.795145:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790670848:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829274.305831:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829274.497942:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829275.329795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829275.521882:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829276.353858:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829276.545914:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829277.377900:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829277.569863:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829278.401839:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829278.593920:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829278.913909:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829278.913920:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829278.913927:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829278.913939:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5915 00000100:00080000:1.0:1787829278.913968:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829278.913976:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829278.914036:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790670976/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829278.914072:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829278.914077:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829278.914080:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829278.914084:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5915 00000100:00080000:1.0:1787829278.914088:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829278.914093:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829278.914102:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829278.914124:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790671104/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829278.914134:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829278.914137:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829278.914139:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829278.914144:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5915 00000100:00080000:1.0:1787829278.914148:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829278.914152:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829278.914157:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829278.914167:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790671232/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829278.914184:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829278.914187:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829278.914189:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829278.914192:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5915 00000100:00080000:1.0:1787829278.914195:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829278.914201:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829278.914209:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829278.914210:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790670976:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787829278.914223:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790671360/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829278.914233:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829278.914236:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829278.914238:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829278.914241:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5915 00000100:00080000:1.0:1787829278.914244:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829278.914248:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829278.914253:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829278.914266:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790671488/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829278.914274:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829278.914277:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829278.914279:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829278.914284:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5915 00000100:00080000:1.0:1787829278.914291:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829278.914297:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829278.914301:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829278.914310:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790671616/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829278.914319:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829278.914323:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829278.914326:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787829278.914335:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829278.914340:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5915 00000100:00080000:1.0:1787829278.914345:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829278.914349:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829278.914354:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829278.914367:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790671744/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829278.914375:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829278.914379:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829278.914381:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829278.914386:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5915 00000100:00080000:1.0:1787829278.914390:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829278.914396:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829278.914403:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829278.914409:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829278/real 1787829278] req@ffff9e6831aa9e00 x1874668790670976/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829278.914422:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790671872/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829278.914448:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790670976/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829278.914512:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829278.914516:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829278.914522:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829278.914540:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787829278.914542:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790670976:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829278.914547:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829278.914564:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5915 00000100:00100000:3.0:1787829278.914566:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790671104:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829278.914574:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829278.914583:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829278.914593:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787829278.914608:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829278.914614:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790672000/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829278.914629:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829278/real 1787829278] req@ffff9e6831aa8f00 x1874668790671104/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829278.914652:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790671104/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829278.914663:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829278.914667:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829278.914671:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790671104:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829278.914805:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790671232:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829278.914830:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829278.914842:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829278/real 1787829278] req@ffff9e6831aaa580 x1874668790671232/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829278.914855:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790671232/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829278.914864:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829278.914868:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829278.914871:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790671232:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829278.914877:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790671360:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829278.914894:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829278.914903:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829278/real 1787829278] req@ffff9e6831aa9680 x1874668790671360/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829278.914914:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790671360/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829278.914930:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829278.914933:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829278.914937:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790671360:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829278.914951:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790671488:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829278.914967:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829278.914975:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829278/real 1787829278] req@ffff9e6831aa8b40 x1874668790671488/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829278.914986:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790671488/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829278.914995:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829278.914998:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829278.915001:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790671488:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829278.915017:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790671616:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829278.915032:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829278.915041:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829278/real 1787829278] req@ffff9e6831aaa940 x1874668790671616/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829278.915051:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790671616/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829278.915062:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829278.915065:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829278.915069:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790671616:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829278.915074:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790671744:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829278.915089:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829278.915098:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829278/real 1787829278] req@ffff9e6831aabc00 x1874668790671744/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829278.915108:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790671744/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829278.915115:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829278.915118:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829278.915121:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790671744:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829278.915127:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790671872:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829278.915142:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829278.915151:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829278/real 1787829278] req@ffff9e6831aa9a40 x1874668790671872/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829278.915161:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790671872/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829278.915170:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829278.915173:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829278.915176:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790671872:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829278.915182:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790672000:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829278.915198:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829278.915207:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829278/real 1787829278] req@ffff9e6831aa8000 x1874668790672000/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829278.915219:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790672000/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829303 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829278.915226:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829278.915229:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829278.915232:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790672000:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829279.425928:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829279.617938:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829280.449824:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829280.641887:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829281.473892:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829281.665939:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829282.497823:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829282.689960:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829283.521827:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829283.713873:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829284.033804:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829284.033815:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829284.033820:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829284.033830:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5920 00000100:00080000:1.0:1787829284.033843:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829284.033850:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829284.033893:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790672128/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829284.033919:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829284.033922:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829284.033929:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829284.033935:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5920 00000100:00080000:1.0:1787829284.033939:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829284.033943:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829284.033956:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829284.033957:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790672128:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787829284.033980:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790672256/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829284.033998:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829284.034002:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829284.034004:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829284.034007:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5920 00000100:00080000:1.0:1787829284.034015:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829284.034019:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:3.0:1787829284.034024:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1787829284.034025:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829284.034040:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790672384/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829284.034050:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829284.034053:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829284.034055:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829284.034059:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5920 00000100:00080000:1.0:1787829284.034062:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829284.034064:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829284/real 1787829284] req@ffff9e6831aaad00 x1874668790672128/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829284.034068:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829284.034080:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829284.034095:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790672512/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829284.034096:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790672128/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829284.034163:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829284.034173:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829284.034182:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790672128:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829284.034187:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829284.034191:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829284.034194:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829284.034199:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5920 00000100:00080000:1.0:1787829284.034202:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829284.034207:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829284.034210:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790672256:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787829284.034220:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787829284.034229:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829284.034238:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790672640/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829284.034244:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829284/real 1787829284] req@ffff9e6831aab480 x1874668790672256/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829284.034298:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829284.034309:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829284.034313:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829284.034324:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5920 00000100:00080000:1.0:1787829284.034328:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829284.034334:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790672256/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829284.034338:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829284.034348:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829284.034364:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787829284.034369:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790672768/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829284.034373:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829284.034389:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790672256:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829284.034401:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829284.034405:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829284.034408:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829284.034408:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790672384:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829284.034423:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5920 00000100:00080000:1.0:1787829284.034427:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829284.034431:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829284.034436:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829284.034446:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829284.034450:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790672896/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829284.034459:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829284.034462:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829284/real 1787829284] req@ffff9e6831aa8780 x1874668790672384/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829284.034467:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829284.034476:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829284.034492:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5920 00000100:00100000:3.0:1787829284.034492:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790672384/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829284.034505:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829284.034507:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829284.034513:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829284.034518:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829284.034523:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790672384:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787829284.034529:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829284.034547:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790672512:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829284.034565:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790673024/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829284.034574:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829284.034590:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829284.034590:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829284/real 1787829284] req@ffff9e6831aaa1c0 x1874668790672512/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829284.034619:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787829284.034620:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790672512/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829284.034626:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829284.034642:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5920 00000100:00080000:1.0:1787829284.034653:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829284.034655:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829284.034661:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:3.0:1787829284.034665:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:1.0:1787829284.034671:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829284.034677:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790672512:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829284.034744:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790672640:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829284.034749:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790673152/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829284.034773:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829284.034788:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829284/real 1787829284] req@ffff9e6831aaad00 x1874668790672640/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829284.034797:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790672640/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829284.034804:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829284.034806:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829284.034809:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790672640:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829284.034814:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790672768:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829284.034825:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829284.034831:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829284/real 1787829284] req@ffff9e6831aa8000 x1874668790672768/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829284.034838:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790672768/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829284.034844:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829284.034846:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829284.034849:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790672768:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829284.034852:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790672896:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829284.034862:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829284.034867:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829284/real 1787829284] req@ffff9e6831aa9a40 x1874668790672896/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829284.034874:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790672896/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829284.034878:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829284.034880:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829284.034882:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790672896:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829284.034885:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790673024:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829284.034895:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829284.034901:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829284/real 1787829284] req@ffff9e6831aabc00 x1874668790673024/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829284.034908:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790673024/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829284.034912:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829284.034914:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829284.034916:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790673024:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829284.034926:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790673152:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829284.034936:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829284.034942:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829284/real 1787829284] req@ffff9e6831aaa940 x1874668790673152/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829284.034949:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790673152/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829284.034954:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829284.034956:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829284.034958:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790673152:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829284.545855:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829284.737939:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829285.569904:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829285.761870:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829286.593786:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829286.785947:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829287.617820:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829287.809889:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829288.641770:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829288.833896:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829289.153794:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829289.153807:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829289.153812:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829289.153825:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5925 00000100:00080000:3.0:1787829289.153838:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829289.153847:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829289.153896:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790673280/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829289.153930:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829289.153934:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829289.153937:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829289.153942:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5925 00000100:00080000:3.0:1787829289.153947:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829289.153951:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829289.153963:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790673280:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787829289.153963:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829289.153979:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790673408/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829289.153989:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829289.153992:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829289.153994:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829289.153997:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5925 00000100:00080000:3.0:1787829289.153999:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829289.154002:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829289.154006:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829289.154014:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790673536/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829289.154021:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829289.154023:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829289.154025:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829289.154027:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5925 00000100:00080000:3.0:1787829289.154030:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829289.154032:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829289.154035:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829289.154042:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829289.154045:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790673664/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829289.154053:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829289.154056:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829289.154058:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829289.154062:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5925 00000100:00080000:3.0:1787829289.154066:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829289.154068:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829289.154071:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829289.154079:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790673792/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829289.154085:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829289.154087:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829289.154089:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829289.154093:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829289/real 1787829289] req@ffff9e680a2d4000 x1874668790673280/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829289.154104:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5925 00000100:00080000:3.0:1787829289.154109:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829289.154113:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790673280/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829289.154114:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829289.154122:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829289.154130:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829289.154135:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829289.154138:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790673920/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829289.154142:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790673280:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829289.154155:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829289.154157:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829289.154158:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829289.154162:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5925 00000100:00080000:3.0:1787829289.154164:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829289.154167:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829289.154169:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829289.154178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790673408:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829289.154179:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790674048/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829289.154193:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829289.154196:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829289.154197:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829289.154200:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5925 00000100:00080000:3.0:1787829289.154203:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829289.154206:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829289.154209:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829289.154215:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829289.154220:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790674176/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829289.154227:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829289.154230:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787829289.154234:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829289/real 1787829289] req@ffff9e680a2d4780 x1874668790673408/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829289.154234:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829289.154242:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5925 00000100:00080000:3.0:1787829289.154248:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829289.154251:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829289.154255:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790673408/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829289.154256:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829289.154268:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829289.154269:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790674304/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829289.154272:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829289.154280:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790673408:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829289.154291:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790673536:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829289.154311:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829289.154318:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829289/real 1787829289] req@ffff9e680a2d43c0 x1874668790673536/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829289.154325:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790673536/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829289.154331:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829289.154333:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829289.154335:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790673536:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829289.154339:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790673664:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829289.154349:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829289.154355:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829289/real 1787829289] req@ffff9e680a2d70c0 x1874668790673664/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829289.154362:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790673664/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829289.154367:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829289.154369:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829289.154371:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790673664:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829289.154374:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790673792:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829289.154383:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829289.154389:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829289/real 1787829289] req@ffff9e680a2d6580 x1874668790673792/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829289.154402:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790673792/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829289.154407:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829289.154409:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829289.154411:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790673792:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829289.154414:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790673920:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829289.154424:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829289.154430:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829289/real 1787829289] req@ffff9e680a2d5a40 x1874668790673920/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829289.154437:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790673920/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829289.154444:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829289.154446:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829289.154447:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790673920:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829289.154462:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790674048:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829289.154472:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829289.154478:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829289/real 1787829289] req@ffff9e680a2d4b40 x1874668790674048/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829289.154486:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790674048/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829289.154495:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829289.154497:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829289.154499:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790674048:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829289.154502:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790674176:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829289.154512:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829289.154517:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829289/real 1787829289] req@ffff9e680a2d7c00 x1874668790674176/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829289.154524:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790674176/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829289.154530:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829289.154531:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829289.154533:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790674176:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829289.154536:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790674304:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829289.154546:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829289.154551:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829289/real 1787829289] req@ffff9e680a2d52c0 x1874668790674304/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829289.154557:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790674304/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829289.154562:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829289.154564:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829289.154567:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790674304:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829289.665832:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829289.858031:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829290.689798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829290.881862:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829291.713813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829291.905871:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829292.737820:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829292.929890:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829293.761851:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829293.953889:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829294.273853:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829294.273862:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829294.273867:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829294.273877:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5930 00000100:00080000:3.0:1787829294.273887:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829294.273893:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829294.273928:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790674432/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829294.273978:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829294.273984:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829294.273987:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829294.273994:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5930 00000100:00080000:3.0:1787829294.274001:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829294.274021:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829294.274036:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829294.274041:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790674432:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829294.274077:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790674560/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829294.274095:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829294.274100:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829294.274103:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829294.274109:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5930 00000100:00080000:3.0:1787829294.274115:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829294.274122:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829294.274131:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829294.274142:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829294.274178:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790674688/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829294.274193:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829294/real 1787829294] req@ffff9e680a2d4f00 x1874668790674432/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829294.274200:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829294.274214:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829294.274219:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829294.274230:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5930 00000100:00080000:3.0:1787829294.274237:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829294.274238:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790674432/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829294.274252:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787829294.274258:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787829294.274264:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829294.274267:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829294.274275:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790674432:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829294.274288:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790674816/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829294.274324:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790674560:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829294.274333:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829294.274339:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829294.274340:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829294.274346:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5930 00000100:00080000:3.0:1787829294.274351:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829294.274356:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829294.274362:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829294.274387:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829294.274398:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790674944/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829294.274413:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829294.274415:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829294/real 1787829294] req@ffff9e680a2d61c0 x1874668790674560/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829294.274425:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829294.274432:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829294.274455:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790674560/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829294.274459:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5930 00000100:00080000:3.0:1787829294.274471:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829294.274473:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829294.274481:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829294.274484:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829294.274488:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790674560:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787829294.274498:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829294.274499:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790674688:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829294.274521:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790675072/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829294.274551:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829294.274555:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829294.274561:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829294.274564:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829294.274572:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829294/real 1787829294] req@ffff9e680a2d6d00 x1874668790674688/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829294.274573:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5930 00000100:00080000:3.0:1787829294.274582:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829294.274592:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829294.274601:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790674688/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829294.274602:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829294.274640:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829294.274647:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829294.274647:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790675200/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829294.274673:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790674688:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829294.274691:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829294.274695:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829294.274697:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829294.274704:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790674816:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829294.274705:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5930 00000100:00080000:3.0:1787829294.274749:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829294.274757:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829294.274764:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829294.274770:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829294.274785:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790675328/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829294.274787:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829294/real 1787829294] req@ffff9e680a2d5e00 x1874668790674816/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829294.274824:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829294.274832:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790674816/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829294.274832:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829294.274837:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829294.274848:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5930 00000100:00080000:3.0:1787829294.274855:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829294.274863:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787829294.274868:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787829294.274869:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829294.274882:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829294.274888:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790675456/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829294.274895:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790674816:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829294.274914:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790674944:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829294.274938:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829294.274970:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829294/real 1787829294] req@ffff9e680a2d7480 x1874668790674944/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829294.274989:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790674944/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829294.275002:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829294.275008:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829294.275029:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790674944:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829294.275037:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790675072:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829294.275064:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829294.275093:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829294/real 1787829294] req@ffff9e680a2d4f00 x1874668790675072/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829294.275103:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790675072/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829294.275111:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829294.275113:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829294.275116:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790675072:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829294.275121:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790675200:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829294.275138:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829294.275167:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829294/real 1787829294] req@ffff9e680a2d52c0 x1874668790675200/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829294.275183:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790675200/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829294.275195:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829294.275200:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829294.275206:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790675200:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829294.275230:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790675328:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829294.275251:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829294.275262:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829294/real 1787829294] req@ffff9e680a2d7c00 x1874668790675328/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829294.275272:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790675328/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829294.275280:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829294.275283:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829294.275286:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790675328:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829294.275290:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790675456:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829294.275303:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829294.275314:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829294/real 1787829294] req@ffff9e680a2d4b40 x1874668790675456/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829294.275330:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790675456/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829294.275341:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829294.275346:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829294.275351:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790675456:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829294.785797:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829294.977857:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829295.809851:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829296.001864:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829296.833878:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829297.025956:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829297.857890:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829298.049841:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829298.881880:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829299.073865:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829299.393788:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829299.393799:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829299.393804:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829299.393817:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5936 00000100:00080000:1.0:1787829299.393829:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829299.393838:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829299.393887:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790675584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829299.393913:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829299.393916:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829299.393918:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829299.393921:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5936 00000100:00080000:1.0:1787829299.393923:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829299.393926:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829299.393936:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829299.393946:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790675712/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829299.393964:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829299.393966:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790675584:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829299.393969:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829299.393974:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829299.393987:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5936 00000100:00080000:1.0:1787829299.393991:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829299.393993:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829299.393996:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829299.394007:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790675840/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829299.394014:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829299.394018:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829299.394019:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829299.394022:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5936 00000100:00080000:1.0:1787829299.394044:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829299.394048:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829299.394052:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829299.394060:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790675968/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829299.394067:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829299.394070:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829299.394071:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829299.394076:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5936 00000100:00080000:1.0:1787829299.394079:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829299.394083:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829299.394088:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829299.394101:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790676096/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829299.394107:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829299.394110:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829299.394113:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829299.394117:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5936 00000100:00080000:1.0:1787829299.394119:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829299.394122:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829299.394125:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829299.394132:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790675712:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829299.394140:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790676224/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829299.394153:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829299.394155:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829299.394157:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829299.394161:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5936 00000100:00080000:1.0:1787829299.394165:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829299.394170:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829299.394173:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829299.394185:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790676352/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829299.394190:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790675840:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829299.394216:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829299.394219:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829299.394222:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829299.394227:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5936 00000100:00080000:1.0:1787829299.394231:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829299.394232:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790675968:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787829299.394242:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829299.394248:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829299.394260:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790676480/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829299.394269:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829299.394272:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790676096:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787829299.394278:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829299.394286:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829299.394293:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5936 00000100:00080000:1.0:1787829299.394296:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829299.394298:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829299.394303:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829299.394315:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790676608/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829299.394327:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790676224:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829299.394364:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790676352:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829299.394394:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790676480:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829299.394430:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790676608:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829299.905789:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829300.097910:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829300.929861:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829301.121942:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829301.953863:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829302.145879:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787829302.465993:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:2.0:1787829302.466030:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787829302.466045:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787829302.466092:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:2.0:1787829302.466109:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787829302.466135:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787829302.466150:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787829302.466159:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787829302.466168:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1787829302.466178:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829299/real 1787829302] req@ffff9e6831aa8000 x1874668790676608/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787829302.466179:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787829302.466193:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1787829302.466241:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790676608/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829302.466257:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829302.466264:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829302.466269:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790676608:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829302.466280:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829299/real 1787829302] req@ffff9e6831aaa940 x1874668790676224/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829302.466295:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790676224/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829302.466307:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829302.466311:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829302.466314:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790676224:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829302.466320:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829299/real 1787829302] req@ffff9e6831aabc00 x1874668790676352/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829302.466329:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790676352/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829302.466334:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829302.466337:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829302.466341:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790676352:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829302.466350:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829299/real 1787829302] req@ffff9e6831aa9a40 x1874668790676480/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829302.466362:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790676480/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829302.466370:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829302.466375:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829302.466378:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790676480:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829302.466386:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829299/real 1787829302] req@ffff9e6831aa9680 x1874668790675712/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829302.466399:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790675712/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829302.466409:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829302.466414:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829302.466418:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790675712:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829302.466427:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829299/real 1787829302] req@ffff9e6831aaa580 x1874668790675840/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829302.466441:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790675840/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829302.466449:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829302.466453:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829302.466457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790675840:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829302.466465:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829299/real 1787829302] req@ffff9e6831aa8f00 x1874668790675968/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829302.466478:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790675968/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829302.466487:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829302.466491:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829302.466494:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790675968:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829302.466502:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829299/real 1787829302] req@ffff9e6831aa9e00 x1874668790676096/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829302.466514:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790676096/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829302.466522:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829302.466527:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829302.466530:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790676096:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829302.466538:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829299/real 1787829302] req@ffff9e6831aa8b40 x1874668790675584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829302.466551:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790675584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829302.466573:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829302.466577:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829302.466580:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790675584:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829302.977860:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829303.169853:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829304.001816:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829304.193841:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829304.513838:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829304.513849:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829304.513854:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829304.513866:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5941 00000100:00080000:0.0:1787829304.513879:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829304.513887:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829304.513927:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790676736/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829304.513957:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829304.513962:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829304.513965:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829304.513970:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5941 00000100:00080000:0.0:1787829304.513975:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829304.513981:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829304.513994:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829304.514002:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790676736:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787829304.514015:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790676864/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829304.514025:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829304.514028:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829304.514029:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829304.514034:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5941 00000100:00080000:0.0:1787829304.514037:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829304.514041:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829304.514047:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829304.514062:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790676992/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829304.514073:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829304.514077:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829304.514080:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829304.514085:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5941 00000100:00080000:0.0:1787829304.514090:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829304.514098:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:3.0:1787829304.514100:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787829304.514106:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829304.514122:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790677120/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829304.514131:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829304.514135:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829304.514138:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829304.514150:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5941 00000100:00080000:0.0:1787829304.514154:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829304.514158:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829304/real 1787829304] req@ffff9e6803e9a580 x1874668790676736/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829304.514163:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829304.514171:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829304.514188:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790677248/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829304.514190:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790676736/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829304.514216:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829304.514221:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829304.514222:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829304.514224:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829304.514232:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829304.514234:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5941 00000100:00080000:0.0:1787829304.514241:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829304.514244:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790676736:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829304.514266:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829304.514273:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829304.514288:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790677376/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829304.514301:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790676864:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829304.514302:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829304.514313:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829304.514315:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829304.514322:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5941 00000100:00080000:0.0:1787829304.514328:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829304.514332:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829304.514338:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829304.514345:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829304.514358:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790677504/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829304.514365:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829304/real 1787829304] req@ffff9e6803e99680 x1874668790676864/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829304.514381:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829304.514387:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787829304.514389:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790676864/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829304.514391:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829304.514404:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5941 00000100:00080000:3.0:1787829304.514408:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829304.514410:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829304.514417:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829304.514420:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829304.514428:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790676864:192.168.203.109@tcp:38:kworker.0 00000008:00100000:0.0:1787829304.514430:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829304.514440:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790676992:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829304.514450:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790677632/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829304.514471:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829304.514481:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829304.514487:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787829304.514491:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829304/real 1787829304] req@ffff9e6803e99e00 x1874668790676992/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829304.514492:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829304.514503:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5941 00000100:00080000:0.0:1787829304.514511:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829304.514520:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829304.514528:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829304.514529:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790676992/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829304.514559:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790677760/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829304.514559:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829304.514570:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829304.514580:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790676992:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829304.514992:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790677120:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829304.515033:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829304.515052:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829304/real 1787829304] req@ffff9e6803e9b480 x1874668790677120/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829304.515069:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790677120/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829304.515079:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829304.515084:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829304.515088:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790677120:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829304.515095:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790677248:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829304.515118:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829304.515131:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829304/real 1787829304] req@ffff9e6803e9b840 x1874668790677248/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829304.515145:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790677248/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829304.515161:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829304.515165:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829304.515169:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790677248:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829304.515590:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790677376:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829304.515613:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829304.515627:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829304/real 1787829304] req@ffff9e6803e9a1c0 x1874668790677376/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829304.515637:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790677376/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829304.515646:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829304.515650:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829304.515653:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790677376:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829304.515658:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790677504:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829304.515673:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829304.515750:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829304/real 1787829304] req@ffff9e6803e98780 x1874668790677504/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829304.515763:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790677504/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829304.515773:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829304.515777:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829304.515781:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790677504:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829304.515788:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790677632:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829304.515810:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829304.515822:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829304/real 1787829304] req@ffff9e6803e98b40 x1874668790677632/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829304.515833:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790677632/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829304.515839:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829304.515841:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829304.515843:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790677632:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829304.515848:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790677760:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829304.515863:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829304.515869:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829304/real 1787829304] req@ffff9e6803e9a940 x1874668790677760/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829304.515877:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790677760/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829304.515882:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829304.515884:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829304.515886:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790677760:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829305.025895:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829305.217838:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829306.049793:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829306.241843:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829307.073797:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829307.265857:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829308.097795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829308.289852:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829309.121813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829309.313907:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829309.633861:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829309.633876:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829309.633883:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829309.633895:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5946 00000100:00080000:1.0:1787829309.633910:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829309.633920:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829309.633987:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790677888/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829309.634024:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829309.634028:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829309.634031:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829309.634036:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5946 00000100:00080000:1.0:1787829309.634040:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829309.634044:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829309.634057:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829309.634072:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790678016/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829309.634083:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829309.634087:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829309.634090:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829309.634094:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5946 00000100:00080000:1.0:1787829309.634098:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829309.634103:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829309.634110:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829309.634124:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790678144/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829309.634137:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829309.634143:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829309.634149:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829309.634150:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790677888:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829309.634166:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5946 00000100:00080000:1.0:1787829309.634171:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829309.634178:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829309.634186:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829309.634204:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790678272/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829309.634216:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829309.634220:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829309.634223:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787829309.634230:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829309.634231:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5946 00000100:00080000:1.0:1787829309.634240:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829309.634246:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829309.634253:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829309.634270:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790678400/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829309.634277:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829309/real 1787829309] req@ffff9e6831aaad00 x1874668790677888/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829309.634309:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829309.634319:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790677888/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829309.634320:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829309.634326:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829309.634345:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829309.634346:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5946 00000100:00080000:3.0:1787829309.634358:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829309.634359:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829309.634364:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790677888:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829309.634370:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829309.634374:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790678016:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787829309.634383:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829309.634395:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829309.634404:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790678528/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829309.634409:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829309/real 1787829309] req@ffff9e6831aaa1c0 x1874668790678016/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829309.634470:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790678016/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829309.634472:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829309.634493:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829309.634495:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829309.634497:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829309.634500:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829309.634524:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5946 00000100:00100000:3.0:1787829309.634525:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790678016:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829309.634542:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829309.634552:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829309.634561:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829309.634566:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790678144:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829309.634579:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790678656/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829309.634591:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829309.634605:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829309.634609:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829309.634612:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829309.634617:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829309/real 1787829309] req@ffff9e6831aa8780 x1874668790678144/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829309.634624:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5946 00000100:00080000:1.0:1787829309.634637:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829309.634645:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790678144/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829309.634646:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829309.634663:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787829309.634664:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829309.634672:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829309.634675:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790678144:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829309.634681:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790678272:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829309.634688:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790678784/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829309.634750:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829309.634765:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829309.634771:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829309.634773:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829309.634779:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829309/real 1787829309] req@ffff9e6831aab480 x1874668790678272/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829309.634783:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5946 00000100:00080000:1.0:1787829309.634792:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829309.634801:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829309.634809:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829309.634809:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790678272/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829309.634828:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787829309.634835:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790678912/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829309.634836:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829309.634845:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790678272:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829309.634867:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790678400:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829309.634895:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829309.634906:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829309/real 1787829309] req@ffff9e6831aa8b40 x1874668790678400/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829309.634914:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790678400/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829309.634924:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829309.634926:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829309.634928:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790678400:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829309.634932:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790678528:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829309.634943:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829309.634948:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829309/real 1787829309] req@ffff9e6831aa9e00 x1874668790678528/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829309.634955:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790678528/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829309.634962:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829309.634964:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829309.634966:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790678528:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829309.634979:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790678656:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829309.634989:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829309.634995:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829309/real 1787829309] req@ffff9e6831aa8f00 x1874668790678656/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829309.635003:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790678656/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829309.635007:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829309.635009:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829309.635012:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790678656:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829309.635016:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790678784:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829309.635026:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829309.635031:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829309/real 1787829309] req@ffff9e6831aaa580 x1874668790678784/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829309.635038:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790678784/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829309.635043:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829309.635045:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829309.635047:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790678784:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829309.635050:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790678912:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829309.635060:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829309.635066:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829309/real 1787829309] req@ffff9e6831aa9680 x1874668790678912/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829309.635072:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790678912/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829309.635077:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829309.635079:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829309.635080:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790678912:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829310.145770:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829310.337951:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829311.169837:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829311.361918:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829312.193748:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829312.385843:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829313.217793:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829313.409836:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829314.241760:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829314.433887:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829314.753815:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829314.753830:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829314.753834:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829314.753842:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5951 00000100:00080000:1.0:1787829314.753852:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829314.753858:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829314.753897:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790679040/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829314.753913:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829314.753916:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829314.753917:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829314.753923:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5951 00000100:00080000:1.0:1787829314.753931:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829314.753936:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829314.753944:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829314.753953:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790679040:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787829314.753963:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790679168/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829314.753973:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829314.753976:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829314.753977:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829314.753980:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5951 00000100:00080000:1.0:1787829314.753984:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829314.753987:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829314.753991:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829314.753999:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790679296/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829314.754007:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829314.754010:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829314.754013:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829314.754020:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5951 00000100:00080000:1.0:1787829314.754023:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829314.754027:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829314.754032:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829314.754041:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790679424/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829314.754048:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829314.754058:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829314.754061:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829314.754064:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829314.754074:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5951 00000100:00080000:1.0:1787829314.754077:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829314.754080:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829314.754084:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829314.754097:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790679552/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829314.754105:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829314/real 1787829314] req@ffff9e6831aa9a40 x1874668790679040/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829314.754110:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829314.754127:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829314.754132:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829314.754151:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5951 00000100:00100000:3.0:1787829314.754156:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790679040/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829314.754164:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829314.754169:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829314.754175:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:3.0:1787829314.754176:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829314.754183:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790679040:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787829314.754190:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829314.754207:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790679680/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829314.754219:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829314.754219:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790679168:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829314.754234:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829314.754237:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829314.754242:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5951 00000100:00080000:1.0:1787829314.754247:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829314.754252:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:3.0:1787829314.754258:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1787829314.754259:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829314.754272:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790679808/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829314.754274:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829314/real 1787829314] req@ffff9e6831aabc00 x1874668790679168/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829314.754342:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829314.754344:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790679168/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829314.754345:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829314.754350:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829314.754367:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5951 00000100:00080000:3.0:1787829314.754373:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829314.754374:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829314.754379:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829314.754383:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829314.754391:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790679168:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787829314.754393:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829314.754404:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790679296:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829314.754412:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790679936/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829314.754422:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829314.754426:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1787829314.754433:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829314.754434:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829314.754446:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5951 00000100:00080000:1.0:1787829314.754451:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829314.754457:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829314/real 1787829314] req@ffff9e6831aaa940 x1874668790679296/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829314.754462:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829314.754483:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829314.754502:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790680064/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829314.754519:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790679296/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829314.754529:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829314.754532:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829314.754536:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790679296:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829314.754541:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790679424:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829314.754560:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829314.754567:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829314/real 1787829314] req@ffff9e6831aa8000 x1874668790679424/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829314.754576:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790679424/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829314.754582:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829314.754584:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829314.754586:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790679424:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829314.754590:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790679552:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829314.754600:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829314.754605:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829314/real 1787829314] req@ffff9e6831aa9680 x1874668790679552/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829314.754613:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790679552/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829314.754622:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829314.754625:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829314.754627:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790679552:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829314.754640:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790679680:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829314.754651:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829314.754656:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829314/real 1787829314] req@ffff9e6831aaa580 x1874668790679680/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829314.754664:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790679680/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829314.754671:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829314.754673:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829314.754675:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790679680:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829314.754679:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790679808:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829314.754689:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829314.754722:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829314/real 1787829314] req@ffff9e6831aa8f00 x1874668790679808/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829314.754731:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790679808/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829314.754737:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829314.754739:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829314.754742:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790679808:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829314.754746:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790679936:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829314.754759:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829314.754766:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829314/real 1787829314] req@ffff9e6831aa9e00 x1874668790679936/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829314.754773:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790679936/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829314.754778:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829314.754780:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829314.754782:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790679936:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829314.754785:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790680064:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829314.754796:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829314.754802:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829314/real 1787829314] req@ffff9e6831aa8b40 x1874668790680064/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829314.754809:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790680064/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829314.754814:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829314.754816:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829314.754818:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790680064:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829315.265803:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829315.457858:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829316.289821:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829316.481896:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829317.313821:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829317.505916:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829318.337828:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829318.529900:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829319.361826:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829319.553850:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829319.873801:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829319.873812:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829319.873816:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829319.873827:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5956 00000100:00080000:1.0:1787829319.873839:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829319.873846:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829319.873897:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790680192/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829319.873932:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829319.873935:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829319.873937:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829319.873940:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5956 00000100:00080000:1.0:1787829319.873944:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829319.873947:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829319.873958:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829319.873970:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790680320/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829319.873978:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829319.873981:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829319.873982:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829319.873985:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5956 00000100:00080000:1.0:1787829319.873989:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829319.873995:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829319.873995:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790680192:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787829319.874000:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829319.874009:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790680448/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829319.874018:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829319.874020:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829319.874022:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829319.874024:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5956 00000100:00080000:1.0:1787829319.874026:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829319.874028:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829319.874032:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829319.874040:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790680576/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829319.874047:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829319.874050:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829319.874052:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829319.874054:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5956 00000100:00080000:1.0:1787829319.874056:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829319.874058:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829319.874067:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829319.874075:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790680704/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829319.874086:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829319.874089:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829319.874091:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787829319.874095:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829319.874098:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5956 00000100:00080000:1.0:1787829319.874102:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829319.874105:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829319.874113:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829319.874122:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790680832/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829319.874128:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829319.874131:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829319.874132:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829319.874134:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5956 00000100:00080000:1.0:1787829319.874137:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829319.874139:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829319.874142:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829319.874155:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790680960/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829319.874156:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829319/real 1787829319] req@ffff9e6831aab480 x1874668790680192/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829319.874177:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829319.874182:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829319.874186:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829319.874192:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5956 00000100:00080000:1.0:1787829319.874198:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829319.874205:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790680192/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829319.874206:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829319.874215:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787829319.874228:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787829319.874235:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790681088/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829319.874235:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829319.874242:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790680192:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829319.874255:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790680320:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829319.874257:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829319.874263:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829319.874266:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829319.874272:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5956 00000100:00080000:1.0:1787829319.874276:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829319.874280:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829319.874287:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787829319.874293:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829319.874303:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790681216/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829319.874310:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829319/real 1787829319] req@ffff9e6831aa8780 x1874668790680320/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829319.874322:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790680320/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829319.874330:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829319.874333:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829319.874336:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790680320:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829319.874362:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790680448:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829319.874375:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829319.874382:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829319/real 1787829319] req@ffff9e6831aaa1c0 x1874668790680448/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829319.874391:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790680448/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829319.874397:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829319.874398:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829319.874400:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790680448:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829319.874404:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790680576:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829319.874416:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829319.874422:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829319/real 1787829319] req@ffff9e6831aaad00 x1874668790680576/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829319.874430:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790680576/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829319.874449:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829319.874452:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829319.874454:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790680576:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829319.874459:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790680704:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829319.874474:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829319.874482:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829319/real 1787829319] req@ffff9e6831aa8b40 x1874668790680704/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829319.874489:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790680704/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829319.874500:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829319.874502:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829319.874504:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790680704:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829319.874508:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790680832:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829319.874518:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829319.874524:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829319/real 1787829319] req@ffff9e6831aa9e00 x1874668790680832/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829319.874531:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790680832/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829319.874539:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829319.874540:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829319.874542:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790680832:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829319.874546:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790680960:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829319.874556:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829319.874563:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829319/real 1787829319] req@ffff9e6831aa8f00 x1874668790680960/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829319.874570:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790680960/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829319.874575:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829319.874576:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829319.874579:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790680960:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829319.874582:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790681088:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829319.874592:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829319.874598:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829319/real 1787829319] req@ffff9e6831aaa580 x1874668790681088/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829319.874604:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790681088/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829319.874609:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829319.874611:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829319.874613:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790681088:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829319.874616:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790681216:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829319.874626:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829319.874631:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829319/real 1787829319] req@ffff9e6831aa9680 x1874668790681216/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829319.874637:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790681216/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829344 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829319.874642:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829319.874645:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829319.874647:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790681216:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829320.385804:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829320.577878:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829321.409813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829321.601999:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829322.433810:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829322.625898:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829323.457795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829323.649885:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829324.481797:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829324.673866:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829324.993819:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829324.993830:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829324.993835:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829324.993847:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5961 00000100:00080000:1.0:1787829324.993859:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829324.993866:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829324.993913:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790681344/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829324.993940:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829324.993944:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829324.993945:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829324.993948:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5961 00000100:00080000:1.0:1787829324.993951:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829324.993953:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829324.993964:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829324.993976:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790681472/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829324.993984:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829324.993987:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829324.993989:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829324.993991:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5961 00000100:00080000:1.0:1787829324.993993:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829324.993998:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829324.994000:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790681344:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787829324.994001:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829324.994011:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790681600/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829324.994018:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829324.994021:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829324.994022:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829324.994025:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5961 00000100:00080000:1.0:1787829324.994027:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829324.994030:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829324.994034:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829324.994043:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790681728/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829324.994053:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829324.994057:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829324.994059:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829324.994063:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5961 00000100:00080000:1.0:1787829324.994076:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829324.994081:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:3.0:1787829324.994087:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1787829324.994090:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829324.994106:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790681856/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829324.994114:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829324.994117:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829324.994118:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829324.994122:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5961 00000100:00080000:1.0:1787829324.994125:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829324.994129:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829324.994133:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829324.994141:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790681984/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829324.994146:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829324/real 1787829324] req@ffff9e6831aa8000 x1874668790681344/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829324.994151:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829324.994158:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829324.994162:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829324.994170:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5961 00000100:00080000:1.0:1787829324.994173:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829324.994175:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829324.994178:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829324.994185:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790681344/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829324.994186:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790682112/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829324.994211:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829324.994216:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829324.994217:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829324.994220:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829324.994221:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5961 00000100:00080000:1.0:1787829324.994226:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829324.994230:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829324.994231:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829324.994237:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829324.994240:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790681344:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787829324.994247:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790682240/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829324.994255:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790681472:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829324.994256:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829324.994264:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829324.994266:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829324.994270:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5961 00000100:00080000:1.0:1787829324.994272:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829324.994275:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829324.994278:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829324.994287:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790682368/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829324.994295:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829324.994316:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829324/real 1787829324] req@ffff9e6831aaa940 x1874668790681472/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829324.994330:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790681472/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829324.994341:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829324.994344:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829324.994348:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790681472:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829324.994377:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790681600:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829324.994408:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829324.994421:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829324/real 1787829324] req@ffff9e6831aabc00 x1874668790681600/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829324.994434:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790681600/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829324.994443:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829324.994446:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829324.994450:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790681600:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829324.994455:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790681728:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829324.994473:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829324.994482:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829324/real 1787829324] req@ffff9e6831aa9a40 x1874668790681728/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829324.994494:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790681728/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829324.994502:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829324.994506:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829324.994509:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790681728:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829324.994514:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790681856:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829324.994530:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829324.994537:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829324/real 1787829324] req@ffff9e6831aa9680 x1874668790681856/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829324.994544:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790681856/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829324.994557:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829324.994560:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829324.994562:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790681856:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829324.994566:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790681984:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829324.994578:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829324.994585:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829324/real 1787829324] req@ffff9e6831aaa580 x1874668790681984/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829324.994592:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790681984/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829324.994599:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829324.994600:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829324.994602:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790681984:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829324.994606:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790682112:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829324.994616:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829324.994622:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829324/real 1787829324] req@ffff9e6831aa8f00 x1874668790682112/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829324.994630:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790682112/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829324.994634:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829324.994636:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829324.994639:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790682112:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829324.994642:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790682240:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829324.994652:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829324.994657:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829324/real 1787829324] req@ffff9e6831aa9e00 x1874668790682240/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829324.994664:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790682240/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829324.994669:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829324.994671:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829324.994673:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790682240:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829324.994676:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790682368:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829324.994686:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829324.994691:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829324/real 1787829324] req@ffff9e6831aa8b40 x1874668790682368/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829324.994722:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790682368/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829349 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829324.994728:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829324.994730:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829324.994732:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790682368:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829325.505789:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829325.697850:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829326.529808:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829326.721865:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829327.553786:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829327.745903:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829328.577800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829328.769872:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829329.601768:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829329.793899:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829330.113792:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829330.113799:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829330.113802:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829330.113811:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5966 00000100:00080000:3.0:1787829330.113822:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829330.113829:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829330.113866:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790682496/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829330.113892:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829330.113895:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829330.113898:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829330.113904:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5966 00000100:00080000:3.0:1787829330.113914:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829330.113920:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790682496:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829330.113922:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829330.113936:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829330.113949:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790682624/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829330.113957:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829330.113960:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829330.113961:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829330.113964:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5966 00000100:00080000:3.0:1787829330.113967:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829330.113970:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829330.113975:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829330.113985:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790682752/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829330.113986:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829330.113999:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829330.114004:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829330.114006:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829330.114011:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5966 00000100:00080000:3.0:1787829330.114015:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829330.114021:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829330.114027:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829330.114028:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829330/real 1787829330] req@ffff9e680a2d5a40 x1874668790682496/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829330.114042:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790682880/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829330.114051:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829330.114053:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790682496/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829330.114054:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829330.114058:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829330.114065:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5966 00000100:00080000:3.0:1787829330.114071:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829330.114076:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829330.114076:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787829330.114086:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:3.0:1787829330.114086:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829330.114092:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790682496:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829330.114105:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790683008/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829330.114115:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829330.114116:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790682624:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829330.114123:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829330.114126:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829330.114130:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5966 00000100:00080000:3.0:1787829330.114133:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829330.114136:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829330.114140:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829330.114145:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829330.114151:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790683136/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829330.114159:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829330/real 1787829330] req@ffff9e680a2d6580 x1874668790682624/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829330.114160:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829330.114171:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829330.114176:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829330.114179:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790682624/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829330.114184:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5966 00000100:00080000:0.0:1787829330.114192:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829330.114194:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829330.114199:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829330.114204:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829330.114205:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790682624:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787829330.114214:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829330.114215:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790682752:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829330.114232:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790683264/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829330.114239:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829330.114247:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829330.114252:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829330/real 1787829330] req@ffff9e680a2d70c0 x1874668790682752/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829330.114252:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829330.114256:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829330.114264:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5966 00000100:00080000:3.0:1787829330.114270:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829330.114272:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790682752/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829330.114277:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829330.114285:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829330.114295:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829330.114298:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790683392/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829330.114301:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829330.114308:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790682752:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829330.114314:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829330.114318:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790682880:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829330.114318:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829330.114321:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829330.114327:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5966 00000100:00080000:3.0:1787829330.114330:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829330.114334:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787829330.114338:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787829330.114339:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829330.114349:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829330/real 1787829330] req@ffff9e680a2d43c0 x1874668790682880/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829330.114352:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790683520/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829330.114389:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790682880/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829330.114398:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829330.114401:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829330.114404:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790682880:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829330.114417:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790683008:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829330.114433:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829330.114440:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829330/real 1787829330] req@ffff9e680a2d4780 x1874668790683008/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829330.114448:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790683008/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829330.114453:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829330.114455:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829330.114458:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790683008:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829330.114462:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790683136:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829330.114472:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829330.114477:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829330/real 1787829330] req@ffff9e680a2d4000 x1874668790683136/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829330.114484:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790683136/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829330.114490:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829330.114492:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829330.114494:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790683136:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829330.114497:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790683264:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829330.114507:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829330.114513:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829330/real 1787829330] req@ffff9e680a2d5a40 x1874668790683264/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829330.114524:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790683264/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829330.114533:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829330.114536:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829330.114539:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790683264:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829330.114546:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790683392:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829330.114566:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829330.114577:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829330/real 1787829330] req@ffff9e680a2d4b40 x1874668790683392/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829330.114586:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790683392/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829330.114591:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829330.114593:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829330.114595:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790683392:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829330.114599:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790683520:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829330.114612:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829330.114619:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829330/real 1787829330] req@ffff9e680a2d7c00 x1874668790683520/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829330.114626:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790683520/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829330.114630:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829330.114632:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829330.114634:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790683520:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829330.625830:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829330.817909:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829331.649773:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829331.841869:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829332.673804:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829332.865884:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829333.697816:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829333.889849:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829334.721818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829334.913868:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829335.233824:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829335.233840:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829335.233846:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829335.233859:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5971 00000100:00080000:3.0:1787829335.233874:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829335.233884:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829335.233933:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790683648/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829335.233965:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829335.233969:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829335.233971:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829335.233977:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5971 00000100:00080000:3.0:1787829335.233982:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829335.233987:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829335.233999:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829335.234000:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790683648:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829335.234017:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790683776/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829335.234030:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829335.234034:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829335.234036:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829335.234040:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5971 00000100:00080000:3.0:1787829335.234044:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829335.234049:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829335.234054:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829335.234067:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790683904/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829335.234077:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829335.234081:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829335.234083:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829335.234092:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829335.234095:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5971 00000100:00080000:3.0:1787829335.234102:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829335.234107:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829335.234114:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829335.234127:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790684032/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829335.234137:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829335.234140:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829335.234142:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829335.234147:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5971 00000100:00080000:3.0:1787829335.234151:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829335.234156:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829335.234162:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829335.234164:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829335/real 1787829335] req@ffff9e680a2d52c0 x1874668790683648/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829335.234176:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790684160/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829335.234194:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829335.234198:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829335.234201:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829335.234204:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790683648/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829335.234209:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5971 00000100:00080000:3.0:1787829335.234216:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829335.234225:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787829335.234228:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787829335.234230:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829335.234237:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829335.234244:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790683648:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829335.234247:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790684288/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829335.234258:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829335.234262:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829335.234264:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829335.234270:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5971 00000100:00080000:3.0:1787829335.234274:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829335.234279:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790683776:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829335.234283:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829335.234291:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829335.234306:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790684416/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829335.234317:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829335.234318:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829335.234324:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829335.234342:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829335.234348:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5971 00000100:00100000:0.0:1787829335.234352:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829335/real 1787829335] req@ffff9e680a2d4f00 x1874668790683776/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829335.234354:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829335.234365:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829335.234374:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829335.234380:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790683776/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829335.234392:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790684544/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829335.234396:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829335.234407:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829335.234412:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829335.234415:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790683776:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829335.234417:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829335.234423:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829335.234428:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790683904:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829335.234432:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5971 00000100:00080000:3.0:1787829335.234439:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829335.234444:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829335.234451:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829335.234463:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829335.234465:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790684672/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829335.234490:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829335/real 1787829335] req@ffff9e680a2d7480 x1874668790683904/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829335.234507:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790683904/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829335.234516:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829335.234520:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829335.234523:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790683904:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829335.234531:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790684032:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829335.234555:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829335.234568:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829335/real 1787829335] req@ffff9e680a2d5e00 x1874668790684032/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829335.234581:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790684032/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829335.234591:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829335.234594:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829335.234598:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790684032:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829335.234605:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790684160:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829335.234626:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829335.234639:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829335/real 1787829335] req@ffff9e680a2d6d00 x1874668790684160/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829335.234651:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790684160/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829335.234661:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829335.234664:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829335.234668:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790684160:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829335.234674:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790684288:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829335.234719:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829335.234734:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829335/real 1787829335] req@ffff9e680a2d61c0 x1874668790684288/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829335.234748:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790684288/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829335.234759:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829335.234763:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829335.234767:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790684288:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829335.234792:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790684416:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829335.234814:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829335.234827:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829335/real 1787829335] req@ffff9e680a2d52c0 x1874668790684416/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829335.234841:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790684416/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829335.234849:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829335.234853:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829335.234856:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790684416:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829335.234862:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790684544:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829335.234883:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829335.234896:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829335/real 1787829335] req@ffff9e680a2d7c00 x1874668790684544/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829335.234909:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790684544/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829335.234924:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829335.234928:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829335.234932:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790684544:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829335.234938:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790684672:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829335.234960:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829335.234972:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829335/real 1787829335] req@ffff9e680a2d4b40 x1874668790684672/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829335.234984:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790684672/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829335.234993:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829335.234996:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829335.235000:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790684672:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829335.745841:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829335.937895:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829336.769789:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829336.961904:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829337.793846:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829337.985905:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829338.817780:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829339.009857:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829339.841836:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829340.033880:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829340.353822:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829340.353836:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829340.353842:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829340.353853:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5977 00000100:00080000:1.0:1787829340.353866:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829340.353877:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829340.353924:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790684800/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829340.353954:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829340.353958:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829340.353960:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829340.353967:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5977 00000100:00080000:1.0:1787829340.353973:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829340.353979:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829340.353991:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829340.353995:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790684800:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787829340.354014:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790684928/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829340.354042:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829340.354047:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829340.354050:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829340.354057:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5977 00000100:00080000:1.0:1787829340.354065:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829340.354070:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829340.354076:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829340.354078:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829340.354107:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790685056/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829340.354120:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829340.354124:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829340.354126:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829340.354130:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5977 00000100:00080000:1.0:1787829340.354135:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829340.354140:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829340.354147:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829340.354162:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790685184/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829340.354172:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829340.354176:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829340.354178:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829340.354183:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5977 00000100:00080000:1.0:1787829340.354188:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829340.354193:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829340.354199:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829340.354214:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790685312/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829340.354225:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829340.354229:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829340.354232:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829340.354236:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5977 00000100:00080000:1.0:1787829340.354240:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829340.354244:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829340.354249:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829340.354260:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790685440/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829340.354270:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829340.354273:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829340.354275:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829340.354278:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5977 00000100:00080000:1.0:1787829340.354281:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829340.354283:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829340.354287:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829340.354311:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790685568/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829340.354318:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829340.354321:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829340.354322:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829340.354327:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5977 00000100:00080000:1.0:1787829340.354330:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829340.354336:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829340.354339:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829340/real 1787829340] req@ffff9e6831aaad00 x1874668790684800/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829340.354342:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829340.354357:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790685696/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829340.354365:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829340.354368:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829340.354369:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829340.354371:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5977 00000100:00080000:1.0:1787829340.354374:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829340.354376:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829340.354379:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829340.354385:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790685824/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829340.354416:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790684800/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829340.354429:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829340.354434:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829340.354439:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790684800:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829340.354465:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790684928:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829340.354486:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829340.354497:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829340/real 1787829340] req@ffff9e6831aaa1c0 x1874668790684928/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829340.354506:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790684928/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829340.354512:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829340.354515:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829340.354517:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790684928:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829340.354522:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790685056:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829340.354534:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829340.354541:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829340/real 1787829340] req@ffff9e6831aa8780 x1874668790685056/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829340.354549:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790685056/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829340.354554:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829340.354556:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829340.354558:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790685056:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829340.354562:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790685184:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829340.354573:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829340.354579:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829340/real 1787829340] req@ffff9e6831aab480 x1874668790685184/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829340.354586:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790685184/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829340.354592:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829340.354593:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829340.354595:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790685184:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829340.354601:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790685312:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829340.354619:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829340.354630:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829340/real 1787829340] req@ffff9e6831aa8b40 x1874668790685312/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829340.354644:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790685312/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829340.354654:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829340.354658:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829340.354661:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790685312:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829340.354666:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790685440:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829340.354682:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829340.354691:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829340/real 1787829340] req@ffff9e6831aa9e00 x1874668790685440/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829340.354721:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790685440/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829340.354729:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829340.354732:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829340.354735:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790685440:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829340.354740:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790685568:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829340.354754:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829340.354762:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829340/real 1787829340] req@ffff9e6831aa8f00 x1874668790685568/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829340.354770:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790685568/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829340.354780:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829340.354784:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829340.354786:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790685568:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829340.354790:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790685696:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829340.354805:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829340.354813:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829340/real 1787829340] req@ffff9e6831aaa580 x1874668790685696/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829340.354821:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790685696/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829340.354827:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829340.354829:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829340.354831:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790685696:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829340.354834:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790685824:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829340.354845:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829340.354851:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829340/real 1787829340] req@ffff9e6831aa9680 x1874668790685824/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829340.354858:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790685824/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829340.354863:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829340.354865:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829340.354867:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790685824:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829340.865782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829341.057832:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829341.889836:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829342.081925:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829342.913795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829343.105866:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829343.937811:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829344.129923:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829344.961787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829345.153926:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829345.473807:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829345.473822:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829345.473827:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829345.473848:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5982 00000100:00080000:3.0:1787829345.473862:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829345.473871:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829345.473927:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790685952/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829345.473963:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829345.473967:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829345.473969:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829345.473974:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5982 00000100:00080000:3.0:1787829345.473979:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829345.473984:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829345.473996:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829345.474004:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790685952:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829345.474013:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790686080/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829345.474026:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829345.474030:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829345.474033:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829345.474038:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5982 00000100:00080000:3.0:1787829345.474042:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829345.474047:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829345.474052:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829345.474063:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790686208/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829345.474074:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829345.474077:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829345.474079:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829345.474083:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5982 00000100:00080000:3.0:1787829345.474087:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829345.474091:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829345.474097:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829345.474100:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829345.474112:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790686336/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829345.474124:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829345.474128:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829345.474130:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829345.474134:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5982 00000100:00080000:3.0:1787829345.474138:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829345.474143:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829345.474149:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829345.474157:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829345/real 1787829345] req@ffff9e680a2d5a40 x1874668790685952/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829345.474164:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790686464/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829345.474178:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829345.474183:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790685952/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829345.474184:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829345.474189:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829345.474199:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5982 00000100:00080000:0.0:1787829345.474201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829345.474204:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829345.474208:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829345.474215:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790685952:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829345.474215:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829345.474225:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829345.474240:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790686592/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829345.474247:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790686080:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829345.474271:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829345.474276:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829345.474278:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829345.474283:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5982 00000100:00080000:3.0:1787829345.474288:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829345.474294:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829345.474301:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829345.474308:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829345.474316:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790686720/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829345.474327:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829345/real 1787829345] req@ffff9e680a2d4000 x1874668790686080/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829345.474328:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829345.474336:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829345.474338:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829345.474345:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5982 00000100:00080000:3.0:1787829345.474353:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829345.474357:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790686080/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829345.474362:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787829345.474372:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787829345.474372:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829345.474378:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829345.474382:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790686080:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829345.474390:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790686208:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829345.474391:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790686848/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829345.474406:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829345.474410:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829345.474412:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829345.474418:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829345.474422:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5982 00000100:00080000:3.0:1787829345.474429:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829345.474434:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829345.474438:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829345/real 1787829345] req@ffff9e680a2d4780 x1874668790686208/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829345.474442:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829345.474465:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790686208/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829345.474466:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790686976/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829345.474488:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829345.474493:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829345.474497:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790686208:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829345.474503:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790686336:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829345.474521:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829345.474531:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829345/real 1787829345] req@ffff9e680a2d43c0 x1874668790686336/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829345.474544:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790686336/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829345.474553:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829345.474557:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829345.474560:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790686336:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829345.474567:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790686464:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829345.474586:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829345.474594:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829345/real 1787829345] req@ffff9e680a2d70c0 x1874668790686464/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829345.474602:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790686464/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829345.474608:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829345.474610:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829345.474613:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790686464:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829345.474628:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790686592:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829345.474639:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829345.474645:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829345/real 1787829345] req@ffff9e680a2d6580 x1874668790686592/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829345.474657:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790686592/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829345.474675:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829345.474678:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829345.474682:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790686592:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829345.474688:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790686720:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829345.474726:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829345.474734:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829345/real 1787829345] req@ffff9e680a2d5a40 x1874668790686720/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829345.474742:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790686720/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829345.474748:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829345.474750:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829345.474752:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790686720:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829345.474756:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790686848:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829345.474767:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829345.474772:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829345/real 1787829345] req@ffff9e680a2d4b40 x1874668790686848/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829345.474780:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790686848/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829345.474785:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829345.474786:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829345.474789:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790686848:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829345.474793:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790686976:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829345.474803:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829345.474808:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829345/real 1787829345] req@ffff9e680a2d7c00 x1874668790686976/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829345.474815:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790686976/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829345.474821:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829345.474822:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829345.474824:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790686976:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829345.985812:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829346.177889:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829347.009775:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829347.201883:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829348.033834:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829348.225892:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829349.057761:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829349.249879:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829350.081825:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829350.273855:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829350.593807:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829350.593819:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829350.593824:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829350.593836:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5987 00000100:00080000:3.0:1787829350.593849:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829350.593856:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829350.593902:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790687104/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829350.593937:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829350.593941:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829350.593942:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829350.593946:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5987 00000100:00080000:3.0:1787829350.593948:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829350.593951:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829350.593960:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829350.593970:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790687232/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829350.593979:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829350.593982:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829350.593984:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829350.593988:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790687104:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829350.593989:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5987 00000100:00080000:3.0:1787829350.593992:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829350.593997:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829350.594003:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829350.594009:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790687360/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829350.594015:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829350.594018:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829350.594020:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829350.594022:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5987 00000100:00080000:3.0:1787829350.594026:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829350.594028:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829350.594032:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829350.594041:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790687488/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829350.594047:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829350.594049:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829350.594051:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829350.594053:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5987 00000100:00080000:3.0:1787829350.594056:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829350.594058:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829350.594061:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829350.594067:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790687616/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829350.594072:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829350.594074:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829350.594076:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829350.594081:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829350.594082:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5987 00000100:00080000:3.0:1787829350.594089:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829350.594094:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829350.594101:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829350.594114:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790687744/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829350.594133:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829350.594136:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829350/real 1787829350] req@ffff9e680a2d52c0 x1874668790687104/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829350.594140:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829350.594144:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829350.594155:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5987 00000100:00100000:0.0:1787829350.594161:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790687104/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829350.594161:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829350.594171:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787829350.594178:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787829350.594180:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829350.594184:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829350.594189:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790687104:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829350.594202:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790687872/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829350.594209:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829350.594212:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829350.594214:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829350.594217:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790687232:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829350.594233:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5987 00000100:00080000:3.0:1787829350.594241:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829350.594245:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829350.594251:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829350.594262:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829350.594264:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790688000/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829350.594275:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829350.594278:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829350/real 1787829350] req@ffff9e680a2d61c0 x1874668790687232/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829350.594279:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829350.594283:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829350.594291:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5987 00000100:00080000:3.0:1787829350.594297:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829350.594301:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790687232/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829350.594306:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829350.594317:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829350.594319:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829350.594325:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829350.594329:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790687232:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829350.594338:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790687360:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829350.594339:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790688128/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829350.594354:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829350.594363:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829350/real 1787829350] req@ffff9e680a2d6d00 x1874668790687360/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829350.594372:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790687360/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829350.594379:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829350.594382:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829350.594384:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790687360:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829350.594388:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790687488:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829350.594399:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829350.594404:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829350/real 1787829350] req@ffff9e680a2d5e00 x1874668790687488/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829350.594411:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790687488/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829350.594418:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829350.594420:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829350.594422:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790687488:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829350.594425:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790687616:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829350.594435:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829350.594440:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829350/real 1787829350] req@ffff9e680a2d7480 x1874668790687616/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829350.594446:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790687616/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829350.594451:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829350.594453:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829350.594455:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790687616:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829350.594458:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790687744:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829350.594469:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829350.594474:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829350/real 1787829350] req@ffff9e680a2d4f00 x1874668790687744/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829350.594481:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790687744/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829350.594487:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829350.594489:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829350.594492:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790687744:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829350.594496:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790687872:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829350.594506:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829350.594512:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829350/real 1787829350] req@ffff9e680a2d7c00 x1874668790687872/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829350.594519:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790687872/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829350.594531:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829350.594533:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829350.594535:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790687872:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829350.594552:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790688000:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829350.594564:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829350.594571:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829350/real 1787829350] req@ffff9e680a2d4b40 x1874668790688000/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829350.594578:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790688000/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829350.594584:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829350.594586:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829350.594587:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790688000:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829350.594591:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790688128:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829350.594601:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829350.594607:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829350/real 1787829350] req@ffff9e680a2d5a40 x1874668790688128/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829350.594615:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790688128/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829350.594626:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829350.594628:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829350.594630:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790688128:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829351.105856:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829351.297905:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829352.129792:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829352.321879:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829353.153869:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829353.345877:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829354.177841:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829354.369928:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829355.201762:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829355.393841:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829355.713792:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829355.713800:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829355.713804:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829355.713812:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5992 00000100:00080000:1.0:1787829355.713821:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829355.713827:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829355.713856:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790688256/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829355.713876:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829355.713879:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829355.713880:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829355.713883:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5992 00000100:00080000:1.0:1787829355.713885:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829355.713888:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829355.713895:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829355.713909:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790688384/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829355.713911:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790688256:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829355.713927:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829355.713931:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829355.713933:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829355.713938:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5992 00000100:00080000:1.0:1787829355.713943:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829355.713948:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829355.713952:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829355.713962:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790688512/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829355.713974:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829355.713979:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829355.713979:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829355.713990:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829355.713994:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5992 00000100:00080000:1.0:1787829355.713998:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829355.714001:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829355.714004:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829355.714014:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829355/real 1787829355] req@ffff9e6831aa9a40 x1874668790688256/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829355.714016:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790688640/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829355.714047:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829355.714050:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829355.714051:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829355.714056:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5992 00000100:00100000:0.0:1787829355.714057:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790688256/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829355.714058:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829355.714064:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829355.714069:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829355.714074:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829355.714080:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829355.714080:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790688768/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829355.714087:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790688256:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829355.714096:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829355.714099:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829355.714101:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829355.714104:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5992 00000100:00080000:1.0:1787829355.714107:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829355.714109:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829355.714113:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829355.714115:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790688384:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829355.714122:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790688896/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829355.714131:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829355.714133:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829355.714134:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829355.714138:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829355.714139:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5992 00000100:00080000:1.0:1787829355.714144:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829355.714147:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829355.714151:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829355.714154:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829355/real 1787829355] req@ffff9e6831aabc00 x1874668790688384/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829355.714162:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790689024/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829355.714169:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790688384/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829355.714172:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829355.714179:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829355.714182:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829355.714202:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5992 00000100:00080000:1.0:1787829355.714208:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829355.714214:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787829355.714215:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787829355.714218:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829355.714221:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829355.714225:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790688384:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829355.714230:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790689152/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829355.714233:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790688512:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829355.714251:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829355.714254:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829355.714256:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829355.714260:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829355.714263:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5992 00000100:00080000:1.0:1787829355.714267:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829355.714270:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829355.714274:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829355.714275:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829355/real 1787829355] req@ffff9e6831aaa940 x1874668790688512/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829355.714284:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790689280/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829355.714287:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790688512/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829355.714305:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829355.714309:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829355.714311:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790688512:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829355.714317:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790688640:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829355.714330:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829355.714337:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829355/real 1787829355] req@ffff9e6831aa8000 x1874668790688640/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829355.714344:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790688640/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829355.714349:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829355.714350:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829355.714353:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790688640:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829355.714357:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790688768:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829355.714366:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829355.714372:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829355/real 1787829355] req@ffff9e6831aa9680 x1874668790688768/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829355.714378:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790688768/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829355.714384:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829355.714386:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829355.714387:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790688768:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829355.714398:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790688896:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829355.714408:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829355.714414:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829355/real 1787829355] req@ffff9e6831aaa580 x1874668790688896/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829355.714421:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790688896/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829355.714427:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829355.714429:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829355.714430:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790688896:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829355.714434:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790689024:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829355.714443:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829355.714449:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829355/real 1787829355] req@ffff9e6831aa8f00 x1874668790689024/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829355.714456:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790689024/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829355.714461:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829355.714463:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829355.714464:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790689024:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829355.714468:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790689152:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829355.714476:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829355.714482:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829355/real 1787829355] req@ffff9e6831aa9e00 x1874668790689152/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829355.714489:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790689152/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829355.714494:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829355.714495:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829355.714497:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790689152:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829355.714501:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790689280:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829355.714510:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829355.714515:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829355/real 1787829355] req@ffff9e6831aa8b40 x1874668790689280/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829355.714521:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790689280/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829355.714530:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829355.714533:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829355.714536:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790689280:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829356.225802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829356.417941:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829357.249851:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829357.441935:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829358.273821:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829358.465928:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829359.297834:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829359.489914:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829360.321811:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829360.513910:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829360.833859:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829360.833867:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829360.833870:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829360.833878:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 5997 00000100:00080000:3.0:1787829360.833887:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829360.833892:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829360.833922:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790689408/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829360.833942:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829360.833944:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829360.833946:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829360.833948:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5997 00000100:00080000:3.0:1787829360.833951:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829360.833954:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829360.833963:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829360.833972:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790689536/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829360.833980:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829360.833984:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787829360.833986:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790689408:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829360.833986:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829360.833995:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5997 00000100:00080000:3.0:1787829360.833999:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829360.834001:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829360.834005:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829360.834015:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790689664/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829360.834022:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829360.834024:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829360.834026:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829360.834028:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5997 00000100:00080000:3.0:1787829360.834031:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829360.834034:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829360.834036:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829360.834043:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790689792/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829360.834049:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829360.834055:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829360.834055:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829360.834060:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829360.834065:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 5997 00000100:00080000:3.0:1787829360.834068:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829360.834071:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829360.834075:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829360.834084:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790689920/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829360.834090:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829360.834093:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829360.834096:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829360.834097:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829360/real 1787829360] req@ffff9e680a2d6580 x1874668790689408/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829360.834102:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5997 00000100:00080000:3.0:1787829360.834108:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829360.834114:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829360.834117:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829360.834123:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790689408/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829360.834128:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790690048/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829360.834139:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829360.834142:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829360.834146:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829360.834163:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787829360.834166:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790689408:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829360.834168:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829360.834176:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5997 00000100:00080000:3.0:1787829360.834179:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829360.834182:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829360.834186:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829360.834195:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790689536:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829360.834198:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790690176/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829360.834208:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829360.834211:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829360.834213:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829360.834216:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5997 00000100:00080000:3.0:1787829360.834219:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829360.834224:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829360.834226:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829360.834232:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829360.834241:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829360/real 1787829360] req@ffff9e680a2d70c0 x1874668790689536/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829360.834243:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790690304/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829360.834269:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829360.834272:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790689536/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829360.834274:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829360.834277:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829360.834285:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 5997 00000100:00080000:3.0:1787829360.834291:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829360.834292:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829360.834298:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787829360.834299:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:3.0:1787829360.834305:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829360.834306:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790689536:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829360.834317:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790690432/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829360.834318:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790689664:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829360.834342:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829360.834350:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829360/real 1787829360] req@ffff9e680a2d43c0 x1874668790689664/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829360.834358:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790689664/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829360.834363:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829360.834365:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829360.834368:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790689664:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829360.834372:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790689792:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829360.834382:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829360.834388:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829360/real 1787829360] req@ffff9e680a2d4780 x1874668790689792/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829360.834395:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790689792/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829360.834400:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829360.834402:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829360.834404:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790689792:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829360.834407:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790689920:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829360.834416:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829360.834422:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829360/real 1787829360] req@ffff9e680a2d4000 x1874668790689920/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829360.834429:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790689920/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829360.834434:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829360.834436:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829360.834438:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790689920:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829360.834441:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790690048:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829360.834451:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829360.834457:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829360/real 1787829360] req@ffff9e680a2d5a40 x1874668790690048/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829360.834464:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790690048/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829360.834470:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829360.834472:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829360.834474:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790690048:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829360.834486:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790690176:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829360.834497:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829360.834502:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829360/real 1787829360] req@ffff9e680a2d4b40 x1874668790690176/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829360.834509:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790690176/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829360.834519:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829360.834521:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829360.834523:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790690176:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829360.834527:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790690304:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829360.834536:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829360.834541:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829360/real 1787829360] req@ffff9e680a2d7c00 x1874668790690304/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829360.834548:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790690304/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829360.834554:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829360.834555:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829360.834557:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790690304:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829360.834560:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790690432:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829360.834569:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829360.834575:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829360/real 1787829360] req@ffff9e680a2d4f00 x1874668790690432/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829360.834581:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790690432/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829360.834586:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829360.834588:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829360.834590:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790690432:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829361.345838:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829361.537931:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829362.369797:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829362.561863:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829363.393857:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829363.585887:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829364.417787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829364.609861:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829365.441794:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829365.633867:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829365.953807:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829365.953819:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829365.953825:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829365.953839:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6002 00000100:00080000:3.0:1787829365.953853:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829365.953861:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829365.953913:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790690560/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829365.953946:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829365.953950:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829365.953953:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829365.953957:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6002 00000100:00080000:3.0:1787829365.953961:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829365.953966:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829365.953976:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829365.953989:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790690688/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829365.953995:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790690560:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829365.954000:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829365.954006:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829365.954008:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829365.954012:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6002 00000100:00080000:3.0:1787829365.954014:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829365.954016:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829365.954019:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829365.954028:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790690816/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829365.954040:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829365.954043:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829365.954044:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829365.954046:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6002 00000100:00080000:3.0:1787829365.954049:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829365.954051:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829365.954055:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829365.954062:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790690944/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829365.954068:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829365.954072:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829365.954073:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829365.954075:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6002 00000100:00080000:3.0:1787829365.954077:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829365.954079:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829365.954082:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829365.954089:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790691072/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829365.954106:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829365.954123:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829365.954126:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829365.954132:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6002 00000100:00080000:3.0:1787829365.954137:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829365.954142:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829365.954148:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829365.954150:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790690688:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829365.954177:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790691200/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829365.954188:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790690816:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829365.954193:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829365.954199:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829365.954201:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829365.954207:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6002 00000100:00100000:0.0:1787829365.954210:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790690944:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829365.954214:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829365.954220:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829365.954224:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829365.954236:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790691328/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829365.954243:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790691072:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829365.954253:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829365.954257:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829365.954260:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829365.954266:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6002 00000100:00080000:3.0:1787829365.954271:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829365.954276:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829365.954281:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790691200:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787829365.954284:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829365.954305:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790691456/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829365.954319:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829365.954321:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790691328:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829365.954323:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829365.954327:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829365.954333:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6002 00000100:00080000:3.0:1787829365.954337:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829365.954339:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829365.954343:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829365.954358:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790691584/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829365.954365:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790691456:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829365.954404:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790691584:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829366.465784:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829366.657881:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829367.489843:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829367.681883:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829368.513791:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829368.705870:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787829369.026075:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:0.0:1787829369.026154:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829369.026179:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787829369.026242:0:2387:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:0.0:1787829369.026258:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:1.0:1787829369.026263:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829365/real 1787829369] req@ffff9e680a2d7480 x1874668790690560/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787829369.026268:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787829369.026281:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787829369.026295:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:1.0:1787829369.026296:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790690560/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787829369.026305:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00080000:1.0:1787829369.026311:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000800:00000100:0.0:1787829369.026315:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00080000:1.0:1787829369.026318:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829369.026323:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790690560:192.168.203.109@tcp:250:kworker.0 00000800:00000100:0.0:1787829369.026330:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:1.0:1787829369.026353:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829365/real 1787829369] req@ffff9e680a2d5a40 x1874668790691584/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829369.026370:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790691584/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829369.026379:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829369.026382:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829369.026385:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790691584:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829369.026390:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829365/real 1787829369] req@ffff9e680a2d4b40 x1874668790691456/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829369.026400:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790691456/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829369.026409:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829369.026414:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829369.026417:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790691456:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829369.026426:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829365/real 1787829369] req@ffff9e680a2d4f00 x1874668790691200/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829369.026440:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790691200/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829369.026451:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829369.026455:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829369.026459:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790691200:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829369.026466:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829365/real 1787829369] req@ffff9e680a2d7c00 x1874668790691328/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829369.026479:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790691328/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829369.026488:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829369.026492:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829369.026495:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790691328:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829369.026503:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829365/real 1787829369] req@ffff9e680a2d5e00 x1874668790690688/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829369.026517:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790690688/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829369.026526:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829369.026530:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829369.026534:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790690688:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829369.026540:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829365/real 1787829369] req@ffff9e680a2d6d00 x1874668790690816/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829369.026552:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790690816/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829369.026560:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829369.026564:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829369.026566:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790690816:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829369.026574:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829365/real 1787829369] req@ffff9e680a2d61c0 x1874668790690944/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829369.026587:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790690944/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829369.026595:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829369.026598:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829369.026602:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790690944:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829369.026609:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829365/real 1787829369] req@ffff9e680a2d52c0 x1874668790691072/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829369.026620:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790691072/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829390 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829369.026627:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829369.026630:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829369.026633:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790691072:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829369.537822:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829369.729912:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829370.561791:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829370.753851:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829371.073830:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829371.073841:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829371.073846:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829371.073860:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6007 00000100:00080000:3.0:1787829371.073882:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829371.073892:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829371.073942:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790691712/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829371.073975:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829371.073979:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829371.073981:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829371.073986:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6007 00000100:00080000:3.0:1787829371.073992:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829371.073997:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829371.074010:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829371.074013:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790691712:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829371.074027:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790691840/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829371.074039:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829371.074043:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829371.074050:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829371.074055:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6007 00000100:00080000:3.0:1787829371.074059:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829371.074063:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829371.074069:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829371.074094:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790691968/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829371.074105:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829371.074112:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787829371.074113:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829371.074115:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829371.074122:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6007 00000100:00080000:3.0:1787829371.074126:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829371.074131:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829371.074137:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829371.074151:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790692096/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829371.074168:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829371.074171:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829371.074181:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829371/real 1787829371] req@ffff9e680a2d4000 x1874668790691712/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829371.074185:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829371.074195:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6007 00000100:00080000:3.0:1787829371.074202:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829371.074207:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829371.074212:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790691712/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829371.074215:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829371.074234:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790692224/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829371.074237:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829371.074248:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829371.074255:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829371.074256:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790691712:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829371.074264:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829371.074268:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829371.074274:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6007 00000100:00080000:3.0:1787829371.074279:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829371.074284:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829371.074289:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829371.074299:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790692352/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829371.074309:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790691840:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829371.074312:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829371.074320:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829371.074322:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829371.074327:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6007 00000100:00080000:3.0:1787829371.074331:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829371.074337:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829371.074343:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829371.074349:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829371.074359:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790692480/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829371.074370:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829371/real 1787829371] req@ffff9e680a2d4780 x1874668790691840/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829371.074372:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829371.074379:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829371.074384:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829371.074393:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6007 00000100:00080000:3.0:1787829371.074398:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829371.074399:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790691840/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829371.074406:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829371.074416:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829371.074419:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829371.074425:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829371.074433:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790692608/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829371.074434:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790691840:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829371.074449:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829371.074454:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790691968:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829371.074456:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829371.074461:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829371.074468:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6007 00000100:00080000:3.0:1787829371.074473:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829371.074478:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829371.074484:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829371.074487:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829371.074501:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790692736/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829371.074506:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829371/real 1787829371] req@ffff9e680a2d43c0 x1874668790691968/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829371.074541:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790691968/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829371.074552:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829371.074557:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829371.074562:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790691968:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829371.074570:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790692096:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829371.074592:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829371.074605:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829371/real 1787829371] req@ffff9e680a2d70c0 x1874668790692096/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829371.074617:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790692096/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829371.074625:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829371.074628:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829371.074631:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790692096:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829371.074638:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790692224:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829371.074657:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829371.074669:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829371/real 1787829371] req@ffff9e680a2d6580 x1874668790692224/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829371.074680:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790692224/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829371.074688:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829371.074691:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829371.074719:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790692224:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829371.074741:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790692352:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829371.074764:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829371.074776:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829371/real 1787829371] req@ffff9e680a2d4000 x1874668790692352/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829371.074789:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790692352/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829371.074800:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829371.074804:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829371.074808:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790692352:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829371.074814:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790692480:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829371.074835:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829371.074846:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829371/real 1787829371] req@ffff9e680a2d52c0 x1874668790692480/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829371.074862:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790692480/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829371.074871:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829371.074875:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829371.074878:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790692480:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829371.074885:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790692608:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829371.074904:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829371.074915:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829371/real 1787829371] req@ffff9e680a2d61c0 x1874668790692608/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829371.074927:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790692608/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829371.074935:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829371.074939:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829371.074943:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790692608:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829371.074950:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790692736:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829371.074969:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829371.074980:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829371/real 1787829371] req@ffff9e680a2d6d00 x1874668790692736/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829371.074991:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790692736/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829371.074998:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829371.075001:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829371.075004:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790692736:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829371.585804:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829371.777916:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829372.609822:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829372.801839:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829373.633781:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829373.825866:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829374.657834:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829374.849854:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829375.681799:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829375.873862:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829376.193808:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829376.193820:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829376.193824:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829376.193836:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6012 00000100:00080000:3.0:1787829376.193848:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829376.193855:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829376.193898:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790692864/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829376.193925:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829376.193929:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829376.193930:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829376.193934:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6012 00000100:00080000:3.0:1787829376.193938:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829376.193942:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829376.193953:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829376.193968:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790692992/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829376.193973:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790692864:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829376.193982:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829376.193988:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829376.193990:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829376.193996:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6012 00000100:00080000:3.0:1787829376.194000:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829376.194004:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829376.194009:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829376.194018:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790693120/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829376.194025:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829376.194029:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829376.194031:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829376.194035:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6012 00000800:00000100:1.0:1787829376.194051:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829376.194056:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829376.194097:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829376/real 1787829376] req@ffff9e680a2d5e00 x1874668790692864/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829376.194116:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790692864/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829376.194125:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829376.194129:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829376.194134:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829376.194135:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790692864:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787829376.194147:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829376.194161:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790693248/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829376.194166:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790692992:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829376.194182:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829376.194186:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829376.194188:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829376.194195:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6012 00000800:00000100:1.0:1787829376.194203:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829376.194214:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829376/real 1787829376] req@ffff9e680a2d7c00 x1874668790692992/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829376.194223:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790692992/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829376.194230:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829376.194233:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829376.194235:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790692992:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829376.194239:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790693120:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829376.194253:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829376.194260:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829376/real 1787829376] req@ffff9e680a2d4f00 x1874668790693120/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829376.194273:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790693120/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829376.194282:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829376.194285:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829376.194289:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790693120:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829376.194309:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790693248:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829376.194323:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829376.194330:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829376.194335:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829376.194343:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829376.194353:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790693376/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829376.194365:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829376/real 1787829376] req@ffff9e680a2d4b40 x1874668790693248/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829376.194382:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790693248/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829376.194396:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829376.194406:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829376.194414:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790693248:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829376.194417:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829376.194425:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829376.194428:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829376.194440:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6012 00000100:00100000:1.0:1787829376.194444:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790693376:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829376.194447:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829376.194456:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829376.194465:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829376.194469:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829376.194488:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829376/real 1787829376] req@ffff9e680a2d5a40 x1874668790693376/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829376.194494:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790693504/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829376.194558:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829376.194561:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829376.194566:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829376.194569:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790693376/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829376.194573:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6012 00000100:00080000:3.0:1787829376.194579:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829376.194588:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829376.194602:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829376.194608:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829376.194616:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790693632/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829376.194617:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829376.194631:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790693376:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829376.194650:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829376.194653:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829376.194655:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829376.194659:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6012 00000100:00080000:3.0:1787829376.194662:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829376.194666:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790693504:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829376.194672:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829376.194683:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829376.194719:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790693760/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829376.194732:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829376.194732:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829376.194740:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829376.194744:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829376.194753:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6012 00000100:00080000:3.0:1787829376.194759:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829376.194766:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829376.194769:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829376/real 1787829376] req@ffff9e680a2d7480 x1874668790693504/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829376.194772:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829376.194791:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790693888/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829376.194802:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790693504/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829376.194825:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829376.194828:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829376.194831:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790693504:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829376.194837:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790693632:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829376.194855:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829376.194864:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829376/real 1787829376] req@ffff9e680a2d4b40 x1874668790693632/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829376.194871:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790693632/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829376.194876:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829376.194879:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829376.194881:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790693632:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829376.194892:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790693760:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829376.194903:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829376.194908:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829376/real 1787829376] req@ffff9e680a2d4f00 x1874668790693760/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829376.194915:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790693760/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829376.194920:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829376.194922:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829376.194925:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790693760:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829376.194929:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790693888:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829376.194939:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829376.194944:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829376/real 1787829376] req@ffff9e680a2d7c00 x1874668790693888/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829376.194952:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790693888/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829376.194957:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829376.194958:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829376.194960:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790693888:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829376.705796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829376.897873:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829377.729798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829377.921850:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829378.753822:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829378.945828:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829379.777799:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829379.969879:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829380.801822:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829380.993917:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829381.313775:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829381.313786:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829381.313791:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829381.313810:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6018 00000100:00080000:3.0:1787829381.313824:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829381.313834:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829381.313885:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790694016/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829381.313915:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829381.313919:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829381.313920:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829381.313924:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6018 00000100:00080000:3.0:1787829381.313928:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829381.313931:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829381.313941:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829381.313955:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790694016:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829381.313955:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790694144/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829381.313969:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829381.313971:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829381.313972:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829381.313975:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6018 00000100:00080000:3.0:1787829381.313977:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829381.313979:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829381.313983:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829381.313991:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790694272/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829381.313998:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829381.314001:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829381.314002:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829381.314017:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6018 00000100:00080000:3.0:1787829381.314022:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829381.314025:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829381.314030:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829381.314039:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790694400/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829381.314047:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829381.314049:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829381.314052:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829381.314055:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829381.314059:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6018 00000100:00080000:3.0:1787829381.314062:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829381.314064:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829381.314068:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829381.314076:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790694528/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829381.314082:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829381.314086:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829381.314087:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829381.314090:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6018 00000100:00080000:3.0:1787829381.314093:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829381.314097:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829381/real 1787829381] req@ffff9e680a2d5e00 x1874668790694016/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829381.314100:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829381.314107:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829381.314120:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790694656/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829381.314124:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790694016/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829381.314151:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829381.314151:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829381.314158:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829381.314162:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790694016:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829381.314164:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829381.314169:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829381.314175:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6018 00000100:00080000:3.0:1787829381.314177:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829381.314180:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829381.314183:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829381.314191:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790694784/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829381.314193:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790694144:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829381.314206:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829381.314209:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829381.314211:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829381.314215:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6018 00000100:00080000:3.0:1787829381.314218:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829381.314222:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829381.314224:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829381.314230:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829381.314237:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829381/real 1787829381] req@ffff9e680a2d6d00 x1874668790694144/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829381.314241:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790694912/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829381.314256:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829381.314257:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790694144/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829381.314265:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829381.314269:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829381.314278:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829381.314278:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6018 00000100:00080000:3.0:1787829381.314285:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829381.314286:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829381.314291:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829381.314292:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790694144:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787829381.314298:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829381.314305:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790694272:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829381.314309:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790695040/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829381.314333:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829381.314348:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829381/real 1787829381] req@ffff9e680a2d61c0 x1874668790694272/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829381.314360:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790694272/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829381.314367:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829381.314369:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829381.314372:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790694272:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829381.314376:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790694400:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829381.314393:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829381.314400:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829381/real 1787829381] req@ffff9e680a2d52c0 x1874668790694400/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829381.314409:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790694400/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829381.314415:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829381.314416:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829381.314418:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790694400:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829381.314422:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790694528:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829381.314432:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829381.314438:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829381/real 1787829381] req@ffff9e680a2d4000 x1874668790694528/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829381.314446:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790694528/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829381.314457:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829381.314459:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829381.314462:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790694528:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829381.314466:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790694656:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829381.314477:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829381.314483:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829381/real 1787829381] req@ffff9e680a2d6580 x1874668790694656/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829381.314491:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790694656/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829381.314503:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829381.314505:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829381.314507:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790694656:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829381.314523:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790694784:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829381.314533:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829381.314539:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829381/real 1787829381] req@ffff9e680a2d70c0 x1874668790694784/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829381.314547:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790694784/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829381.314552:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829381.314553:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829381.314555:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790694784:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829381.314559:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790694912:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829381.314569:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829381.314575:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829381/real 1787829381] req@ffff9e680a2d43c0 x1874668790694912/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829381.314582:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790694912/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829381.314587:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829381.314589:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829381.314592:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790694912:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829381.314595:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790695040:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829381.314604:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829381.314610:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829381/real 1787829381] req@ffff9e680a2d4780 x1874668790695040/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829381.314616:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790695040/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829381.314620:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829381.314622:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829381.314624:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790695040:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829381.825843:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829382.017880:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829382.849843:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829383.041857:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829383.873796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829384.065878:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829384.897814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829385.089860:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829385.921811:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829386.113901:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829386.433787:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829386.433803:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829386.433809:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829386.433822:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6023 00000100:00080000:3.0:1787829386.433837:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829386.433846:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829386.433898:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790695168/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829386.433933:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829386.433937:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829386.433939:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829386.433944:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6023 00000100:00080000:3.0:1787829386.433948:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829386.433952:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829386.433964:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829386.433989:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790695296/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829386.434000:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829386.434003:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829386.434004:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829386.434009:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6023 00000100:00080000:3.0:1787829386.434018:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829386.434019:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790695168:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829386.434026:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829386.434037:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829386.434048:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790695424/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829386.434055:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829386.434058:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829386.434059:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829386.434063:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6023 00000100:00080000:3.0:1787829386.434065:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829386.434068:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829386.434071:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829386.434079:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790695552/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829386.434086:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829386.434088:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829386.434089:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829386.434092:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6023 00000100:00080000:3.0:1787829386.434096:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829386.434100:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829386.434105:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829386.434113:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829386.434118:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790695680/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829386.434127:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829386.434130:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829386.434132:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829386.434137:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6023 00000100:00080000:3.0:1787829386.434139:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829386.434142:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829386.434146:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829386.434159:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790695808/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829386.434167:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829386/real 1787829386] req@ffff9e680a2d4780 x1874668790695168/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829386.434169:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829386.434179:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829386.434182:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829386.434195:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6023 00000100:00080000:3.0:1787829386.434204:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829386.434210:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829386.434215:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829386.434217:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790695168/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829386.434229:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790695936/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829386.434234:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829386.434251:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829386.434265:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829386.434268:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790695168:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829386.434270:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829386.434276:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829386.434286:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6023 00000100:00080000:3.0:1787829386.434290:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829386.434295:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829386.434302:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790695296:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787829386.434305:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829386.434329:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829386.434331:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790696064/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829386.434341:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829386/real 1787829386] req@ffff9e680a2d43c0 x1874668790695296/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829386.434353:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829386.434358:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790695296/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829386.434363:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829386.434372:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829386.434378:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829386.434385:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6023 00000100:00080000:1.0:1787829386.434386:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829386.434399:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790695296:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829386.434400:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829386.434411:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829386.434419:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829386.434426:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790695424:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829386.434439:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790696192/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829386.434461:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829386.434476:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829386/real 1787829386] req@ffff9e680a2d70c0 x1874668790695424/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829386.434488:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790695424/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829386.434498:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829386.434501:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829386.434505:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790695424:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829386.434512:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790695552:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829386.434536:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829386.434549:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829386/real 1787829386] req@ffff9e680a2d6580 x1874668790695552/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829386.434569:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790695552/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829386.434578:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829386.434581:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829386.434585:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790695552:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829386.434591:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790695680:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829386.434614:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829386.434636:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829386/real 1787829386] req@ffff9e680a2d4000 x1874668790695680/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829386.434647:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790695680/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829386.434653:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829386.434655:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829386.434658:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790695680:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829386.434662:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790695808:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829386.434678:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829386.434685:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829386/real 1787829386] req@ffff9e680a2d52c0 x1874668790695808/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829386.434692:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790695808/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829386.434749:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829386.434753:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829386.434757:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790695808:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829386.434763:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790695936:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829386.434785:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829386.434797:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829386/real 1787829386] req@ffff9e680a2d61c0 x1874668790695936/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829386.434808:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790695936/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829386.434816:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829386.434820:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829386.434823:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790695936:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829386.434847:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790696064:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829386.434867:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829386.434879:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829386/real 1787829386] req@ffff9e680a2d6d00 x1874668790696064/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829386.434891:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790696064/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829386.434899:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829386.434903:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829386.434907:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790696064:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829386.434914:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790696192:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829386.434935:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829386.434946:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829386/real 1787829386] req@ffff9e680a2d5e00 x1874668790696192/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829386.434954:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790696192/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829386.434960:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829386.434962:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829386.434964:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790696192:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829386.945805:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829387.137853:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829387.969781:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829388.161844:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829388.993850:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829389.185974:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829390.017799:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829390.209863:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829391.041803:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829391.233903:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829391.553878:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829391.553889:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829391.553893:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829391.553904:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6028 00000100:00080000:1.0:1787829391.553915:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829391.553923:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829391.553973:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790696320/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829391.554009:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829391.554013:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829391.554015:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829391.554020:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6028 00000100:00080000:1.0:1787829391.554025:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829391.554029:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829391.554039:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829391.554055:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790696448/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829391.554066:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790696320:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829391.554076:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829391.554081:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829391.554083:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829391.554089:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6028 00000100:00080000:1.0:1787829391.554094:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829391.554098:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829391.554104:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829391.554118:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790696576/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829391.554129:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829391.554134:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829391.554136:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829391.554140:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6028 00000100:00080000:1.0:1787829391.554155:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829391.554161:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787829391.554165:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1787829391.554171:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829391.554194:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790696704/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829391.554207:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829391.554211:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829391.554213:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829391.554220:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829391/real 1787829391] req@ffff9e6831aab480 x1874668790696320/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829391.554221:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6028 00000100:00080000:1.0:1787829391.554234:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829391.554243:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829391.554249:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790696320/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829391.554253:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829391.554271:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829391.554277:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829391.554284:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790696832/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829391.554285:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790696320:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829391.554307:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829391.554311:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829391.554314:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829391.554319:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6028 00000100:00100000:0.0:1787829391.554323:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790696448:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829391.554327:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829391.554336:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829391.554343:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829391.554357:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829391.554360:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790696960/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829391.554375:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829391.554381:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829391/real 1787829391] req@ffff9e6831aa8780 x1874668790696448/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829391.554382:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829391.554386:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829391.554409:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6028 00000100:00100000:0.0:1787829391.554416:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790696448/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829391.554416:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829391.554431:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787829391.554432:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829391.554442:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:1.0:1787829391.554446:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829391.554449:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790696448:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829391.554463:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790696576:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829391.554473:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790697088/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829391.554484:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829391.554487:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1787829391.554497:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829391.554499:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829391.554508:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829391/real 1787829391] req@ffff9e6831aaa1c0 x1874668790696576/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829391.554513:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6028 00000100:00080000:1.0:1787829391.554525:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829391.554531:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790696576/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829391.554536:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829391.554552:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829391.554554:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829391.554564:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829391.554568:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790696576:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829391.554580:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790696704:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829391.554581:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790697216/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829391.554601:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829391.554603:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829391.554610:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829391.554613:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829391.554622:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829391/real 1787829391] req@ffff9e6831aaad00 x1874668790696704/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829391.554623:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6028 00000100:00080000:1.0:1787829391.554632:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829391.554644:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829391.554654:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829391.554668:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790696704/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829391.554672:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790697344/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829391.554727:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829391.554735:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829391.554740:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790696704:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829391.554750:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790696832:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829391.554784:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829391.554803:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829391/real 1787829391] req@ffff9e6831aa8b40 x1874668790696832/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829391.554817:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790696832/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829391.554835:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829391.554840:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829391.554842:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790696832:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829391.554859:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790696960:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829391.554877:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829391.554885:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829391/real 1787829391] req@ffff9e6831aa9e00 x1874668790696960/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829391.554893:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790696960/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829391.554900:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829391.554903:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829391.554905:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790696960:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829391.554909:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790697088:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829391.554920:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829391.554926:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829391/real 1787829391] req@ffff9e6831aa8f00 x1874668790697088/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829391.554950:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790697088/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829391.554956:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829391.554958:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829391.554961:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790697088:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829391.554966:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790697216:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829391.554983:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829391.554990:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829391/real 1787829391] req@ffff9e6831aaa580 x1874668790697216/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829391.554998:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790697216/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829391.555003:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829391.555005:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829391.555007:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790697216:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829391.555011:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790697344:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829391.555022:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829391.555028:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829391/real 1787829391] req@ffff9e6831aa9680 x1874668790697344/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829391.555035:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790697344/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829391.555040:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829391.555042:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829391.555044:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790697344:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829392.065832:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829392.257857:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829393.089819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829393.281857:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829394.113812:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829394.305872:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829395.137806:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829395.329871:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829396.161838:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829396.353879:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829396.673811:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829396.673822:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829396.673826:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829396.673837:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6033 00000100:00080000:3.0:1787829396.673849:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829396.673857:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829396.673914:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790697472/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829396.673941:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829396.673944:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829396.673946:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829396.673953:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6033 00000100:00080000:3.0:1787829396.673956:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829396.673960:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829396.673970:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829396.673980:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790697600/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829396.673988:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829396.673991:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829396.673993:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829396.673999:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790697472:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829396.673999:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6033 00000100:00080000:3.0:1787829396.674003:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829396.674008:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829396.674012:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829396.674021:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790697728/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829396.674028:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829396.674030:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829396.674031:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829396.674034:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6033 00000100:00080000:3.0:1787829396.674036:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829396.674039:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829396.674043:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829396.674051:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790697856/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829396.674057:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829396.674059:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829396.674060:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829396.674063:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6033 00000100:00080000:3.0:1787829396.674066:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829396.674068:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829396.674071:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829396.674078:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790697984/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829396.674085:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829396.674089:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829396.674090:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829396.674095:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829396.674099:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6033 00000100:00080000:3.0:1787829396.674102:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829396.674106:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829396.674109:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829396.674125:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790698112/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829396.674132:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829396.674134:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829396.674135:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829396.674139:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6033 00000100:00100000:0.0:1787829396.674140:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829396/real 1787829396] req@ffff9e680a2d7c00 x1874668790697472/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829396.674142:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829396.674147:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829396.674153:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829396.674165:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790698240/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829396.674167:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790697472/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829396.674185:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829396.674188:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829396.674189:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829396.674191:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829396.674194:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6033 00000100:00080000:3.0:1787829396.674197:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829396.674198:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829396.674202:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829396.674206:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790697472:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787829396.674208:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829396.674219:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790698368/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829396.674226:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829396.674229:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829396.674231:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829396.674234:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790697600:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829396.674237:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6033 00000100:00080000:3.0:1787829396.674242:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829396.674247:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829396.674252:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829396.674266:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829396.674266:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790698496/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829396.674280:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829396/real 1787829396] req@ffff9e680a2d4f00 x1874668790697600/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829396.674292:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790697600/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829396.674303:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829396.674307:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829396.674311:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790697600:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829396.674318:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790697728:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829396.674349:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829396.674362:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829396/real 1787829396] req@ffff9e680a2d4b40 x1874668790697728/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829396.674375:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790697728/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829396.674391:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829396.674395:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829396.674399:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790697728:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829396.674405:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790697856:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829396.674423:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829396.674431:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829396/real 1787829396] req@ffff9e680a2d7480 x1874668790697856/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829396.674439:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790697856/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829396.674449:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829396.674452:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829396.674455:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790697856:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829396.674462:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790697984:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829396.674483:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829396.674495:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829396/real 1787829396] req@ffff9e680a2d5a40 x1874668790697984/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829396.674504:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790697984/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829396.674509:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829396.674511:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829396.674514:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790697984:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829396.674518:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790698112:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829396.674532:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829396.674539:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829396/real 1787829396] req@ffff9e680a2d5e00 x1874668790698112/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829396.674547:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790698112/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829396.674554:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829396.674556:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829396.674558:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790698112:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829396.674561:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790698240:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829396.674571:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829396.674585:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829396/real 1787829396] req@ffff9e680a2d6d00 x1874668790698240/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829396.674596:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790698240/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829396.674605:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829396.674608:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829396.674611:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790698240:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829396.674618:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790698368:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829396.674639:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829396.674651:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829396/real 1787829396] req@ffff9e680a2d61c0 x1874668790698368/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829396.674664:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790698368/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829396.674674:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829396.674677:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829396.674681:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790698368:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829396.674746:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790698496:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829396.674771:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829396.674779:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829396/real 1787829396] req@ffff9e680a2d52c0 x1874668790698496/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829396.674787:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790698496/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829396.674793:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829396.674795:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829396.674797:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790698496:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829397.185782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829397.377873:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829398.209788:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829398.401839:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829399.233813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829399.425911:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829400.257824:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829400.449907:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829401.281824:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829401.473859:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829401.793818:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829401.793830:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829401.793834:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829401.793844:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6038 00000100:00080000:0.0:1787829401.793856:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829401.793874:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829401.793920:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790698624/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829401.793949:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829401.793952:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829401.793954:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829401.793957:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6038 00000100:00080000:0.0:1787829401.793960:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829401.793962:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829401.793973:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829401.793983:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790698752/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829401.793991:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829401.793994:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829401.793996:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829401.793999:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6038 00000100:00080000:0.0:1787829401.794000:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829401.794003:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829401.794006:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829401.794010:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790698624:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787829401.794015:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790698880/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829401.794024:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829401.794027:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829401.794029:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829401.794032:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6038 00000100:00080000:0.0:1787829401.794034:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829401.794037:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829401.794041:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829401.794049:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790699008/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829401.794056:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829401.794059:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829401.794060:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829401.794063:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6038 00000100:00080000:0.0:1787829401.794084:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829401.794086:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829401.794089:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829401.794095:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829401.794100:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790699136/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829401.794107:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829401.794110:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829401.794112:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829401.794115:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6038 00000100:00080000:0.0:1787829401.794120:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829401.794122:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829401.794125:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829401.794133:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790699264/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829401.794141:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829401.794145:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787829401.794145:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829401/real 1787829401] req@ffff9e6803e9bc00 x1874668790698624/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829401.794149:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829401.794157:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6038 00000100:00080000:0.0:1787829401.794161:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829401.794165:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829401.794168:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829401.794173:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790698624/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829401.794178:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790699392/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829401.794194:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829401.794195:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829401.794199:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829401.794204:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829401.794204:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829401.794210:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790698624:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829401.794211:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6038 00000100:00080000:0.0:1787829401.794217:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829401.794221:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829401.794224:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790698752:192.168.203.109@tcp:38:kworker.0 00000008:00100000:0.0:1787829401.794226:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829401.794238:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790699520/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829401.794247:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829401.794249:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829401.794252:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829401.794255:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829401.794258:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6038 00000100:00080000:0.0:1787829401.794262:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829401.794263:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829401/real 1787829401] req@ffff9e6803e992c0 x1874668790698752/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829401.794267:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829401.794274:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829401.794287:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790699648/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829401.794292:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790698752/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829401.794307:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829401.794318:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829401.794325:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790698752:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829401.794354:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790698880:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829401.794371:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829401.794379:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829401/real 1787829401] req@ffff9e6803e98f00 x1874668790698880/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829401.794387:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790698880/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829401.794393:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829401.794395:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829401.794397:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790698880:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829401.794401:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790699008:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829401.794411:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829401.794416:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829401/real 1787829401] req@ffff9e6803e9ad00 x1874668790699008/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829401.794423:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790699008/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829401.794428:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829401.794430:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829401.794432:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790699008:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829401.794435:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790699136:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829401.794445:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829401.794451:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829401/real 1787829401] req@ffff9e6803e9a940 x1874668790699136/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829401.794459:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790699136/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829401.794464:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829401.794466:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829401.794467:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790699136:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829401.794471:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790699264:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829401.794481:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829401.794488:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829401/real 1787829401] req@ffff9e6803e98b40 x1874668790699264/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829401.794495:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790699264/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829401.794501:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829401.794503:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829401.794505:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790699264:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829401.794508:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790699392:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829401.794517:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829401.794523:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829401/real 1787829401] req@ffff9e6803e98780 x1874668790699392/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829401.794529:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790699392/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829401.794535:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829401.794537:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829401.794539:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790699392:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829401.794543:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790699520:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829401.794552:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829401.794558:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829401/real 1787829401] req@ffff9e6803e9a1c0 x1874668790699520/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829401.794564:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790699520/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829401.794569:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829401.794571:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829401.794573:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790699520:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829401.794577:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790699648:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829401.794586:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829401.794591:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829401/real 1787829401] req@ffff9e6803e9b840 x1874668790699648/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829401.794597:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790699648/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829401.794603:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829401.794605:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829401.794607:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790699648:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829402.305834:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829402.497852:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829403.329807:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829403.521869:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829404.353834:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829404.545865:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829405.377823:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829405.569894:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829406.401825:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829406.593882:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829406.913880:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829406.913896:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829406.913902:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829406.913917:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6043 00000100:00080000:0.0:1787829406.913934:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829406.913944:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829406.913999:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790699776/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829406.914034:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829406.914039:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829406.914041:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829406.914045:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6043 00000100:00080000:0.0:1787829406.914050:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829406.914054:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829406.914067:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829406.914083:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790699904/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829406.914098:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790699776:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829406.914105:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829406.914113:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829406.914115:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829406.914121:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6043 00000100:00080000:0.0:1787829406.914126:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829406.914131:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829406.914137:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829406.914151:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790700032/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829406.914162:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829406.914166:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829406.914168:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829406.914172:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6043 00000100:00080000:0.0:1787829406.914177:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829406.914182:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1787829406.914187:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787829406.914192:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829406.914208:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790700160/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829406.914218:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829406.914234:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829406.914237:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829406.914243:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829406/real 1787829406] req@ffff9e6803e9b480 x1874668790699776/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829406.914246:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6043 00000100:00080000:0.0:1787829406.914255:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829406.914263:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829406.914271:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829406.914286:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790699776/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829406.914288:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790700288/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829406.914321:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829406.914321:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829406.914329:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829406.914333:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829406.914334:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829406.914343:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790699776:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829406.914344:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6043 00000100:00080000:0.0:1787829406.914351:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829406.914357:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829406.914366:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829406.914383:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790700416/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829406.914396:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829406.914400:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790699904:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829406.914401:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829406.914406:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829406.914413:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6043 00000100:00080000:0.0:1787829406.914418:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829406.914423:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829406.914430:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829406.914435:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829406.914446:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790700544/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829406.914452:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829406/real 1787829406] req@ffff9e6803e99e00 x1874668790699904/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829406.914467:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829406.914473:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790699904/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829406.914474:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829406.914480:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829406.914488:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829406.914490:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6043 00000100:00080000:1.0:1787829406.914497:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829406.914498:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829406.914505:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790699904:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829406.914508:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829406.914520:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829406.914523:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790700032:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829406.914537:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790700672/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829406.914550:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829406.914556:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829406.914557:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829406.914561:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829406.914568:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6043 00000100:00100000:1.0:1787829406.914573:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829406/real 1787829406] req@ffff9e6803e99680 x1874668790700032/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829406.914575:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829406.914585:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829406.914590:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790700032/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829406.914594:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829406.914607:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829406.914614:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829406.914615:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790700800/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829406.914620:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790700032:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829406.914642:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790700160:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829406.914671:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829406.914685:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829406/real 1787829406] req@ffff9e6803e9a580 x1874668790700160/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829406.914734:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790700160/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829406.914741:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829406.914744:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829406.914746:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790700160:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829406.914751:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790700288:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829406.914768:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829406.914776:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829406/real 1787829406] req@ffff9e6803e9b840 x1874668790700288/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829406.914789:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790700288/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829406.914795:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829406.914797:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829406.914799:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790700288:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829406.914812:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790700416:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829406.914824:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829406.914850:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829406/real 1787829406] req@ffff9e6803e9a1c0 x1874668790700416/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829406.914859:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790700416/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829406.914866:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829406.914868:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829406.914870:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790700416:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829406.914875:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790700544:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829406.914888:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829406.914895:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829406/real 1787829406] req@ffff9e6803e98780 x1874668790700544/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829406.914902:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790700544/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829406.914920:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829406.914922:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829406.914924:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790700544:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829406.914929:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790700672:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829406.914940:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829406.914947:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829406/real 1787829406] req@ffff9e6803e98b40 x1874668790700672/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829406.914954:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790700672/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829406.914959:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829406.914961:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829406.914963:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790700672:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829406.914973:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790700800:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829406.914993:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829406.915005:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829406/real 1787829406] req@ffff9e6803e9a940 x1874668790700800/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829406.915017:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790700800/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829431 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829406.915026:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829406.915030:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829406.915033:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790700800:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829407.425780:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829407.617897:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829408.449839:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829408.641843:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829409.473825:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829409.665900:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829410.497829:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829410.689856:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829411.521813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829411.713914:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829412.033893:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829412.033903:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829412.033907:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829412.033916:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6048 00000100:00080000:0.0:1787829412.033926:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829412.033935:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829412.033971:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790700928/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829412.034010:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829412.034013:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829412.034014:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829412.034018:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6048 00000100:00080000:0.0:1787829412.034021:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829412.034027:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829412.034037:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829412.034051:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790701056/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829412.034062:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790700928:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829412.034067:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829412.034078:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829412.034080:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829412.034085:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6048 00000100:00080000:0.0:1787829412.034088:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829412.034091:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829412.034093:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829412.034107:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790701184/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829412.034114:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829412.034118:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829412.034120:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829412.034123:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6048 00000100:00080000:0.0:1787829412.034126:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829412.034128:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829412.034134:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829412.034144:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790701312/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829412.034152:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829412.034155:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829412.034156:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829412.034159:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6048 00000100:00080000:0.0:1787829412.034160:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829412.034162:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829412.034165:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829412.034174:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790701440/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829412.034179:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829412.034181:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829412.034182:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829412.034185:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6048 00000100:00080000:0.0:1787829412.034187:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829412.034189:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829412.034192:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829412.034198:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790701568/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829412.034209:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829412.034211:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829412.034212:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829412.034214:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6048 00000100:00080000:0.0:1787829412.034216:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829412.034218:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829412.034220:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829412.034226:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790701696/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829412.034230:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829412.034232:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829412.034233:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829412.034235:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6048 00000100:00080000:0.0:1787829412.034237:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829412.034240:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829412.034242:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829412.034247:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790701824/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829412.034253:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829412.034255:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829412.034255:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829412.034258:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6048 00000100:00080000:0.0:1787829412.034260:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829412.034263:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829412.034265:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829412.034270:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790701952/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829412.034378:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829412.034487:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829412/real 1787829412] req@ffff9e6803e9ad00 x1874668790700928/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829412.034506:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790700928/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829412.034516:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829412.034520:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829412.034525:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790700928:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829412.034563:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790701056:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829412.034595:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829412.034612:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829412/real 1787829412] req@ffff9e6803e98f00 x1874668790701056/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829412.034626:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790701056/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829412.034635:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829412.034638:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829412.034640:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790701056:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829412.034645:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790701184:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829412.034663:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829412.034671:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829412/real 1787829412] req@ffff9e6803e992c0 x1874668790701184/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829412.034678:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790701184/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829412.034684:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829412.034686:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829412.034689:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790701184:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829412.034693:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790701312:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829412.034737:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829412.034750:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829412/real 1787829412] req@ffff9e6803e9bc00 x1874668790701312/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829412.034767:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790701312/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829412.034775:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829412.034777:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829412.034781:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790701312:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829412.034789:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790701440:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829412.034821:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829412.034829:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829412/real 1787829412] req@ffff9e6803e9a940 x1874668790701440/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829412.034838:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790701440/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829412.034844:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829412.034846:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829412.034848:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790701440:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829412.034852:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790701568:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829412.034863:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829412.034868:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829412/real 1787829412] req@ffff9e6803e98b40 x1874668790701568/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829412.034875:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790701568/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829412.034883:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829412.034885:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829412.034887:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790701568:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829412.034891:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790701696:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829412.034901:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829412.034907:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829412/real 1787829412] req@ffff9e6803e98780 x1874668790701696/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829412.034914:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790701696/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829412.034919:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829412.034921:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829412.034923:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790701696:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829412.034926:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790701824:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829412.034937:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829412.034943:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829412/real 1787829412] req@ffff9e6803e9a1c0 x1874668790701824/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829412.034949:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790701824/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829412.034954:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829412.034956:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829412.034959:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790701824:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829412.034963:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790701952:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829412.034974:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829412.034980:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829412/real 1787829412] req@ffff9e6803e9b840 x1874668790701952/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829412.034987:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790701952/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829412.034992:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829412.034994:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829412.034996:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790701952:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829412.545855:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829412.737868:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829413.569840:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829413.761864:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829414.593816:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829414.785893:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829415.617827:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829415.809891:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829416.641818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829416.833917:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829417.153823:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829417.153834:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829417.153839:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829417.153851:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6053 00000100:00080000:0.0:1787829417.153863:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829417.153871:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829417.153924:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790702080/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829417.153953:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829417.153957:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829417.153959:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829417.153962:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6053 00000100:00080000:0.0:1787829417.153966:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829417.153970:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829417.153981:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829417.153989:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790702080:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787829417.153996:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790702208/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829417.154005:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829417.154010:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829417.154012:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829417.154015:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6053 00000100:00080000:0.0:1787829417.154017:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829417.154019:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829417.154022:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829417.154033:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790702336/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829417.154039:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829417.154042:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829417.154044:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829417.154046:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6053 00000100:00080000:0.0:1787829417.154049:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829417.154052:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829417.154056:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829417.154064:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790702464/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829417.154071:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829417.154077:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829417.154080:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829417.154082:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829417.154086:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6053 00000100:00080000:0.0:1787829417.154090:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829417.154092:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829417.154096:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829417.154106:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790702592/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829417.154112:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829417.154115:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829417.154117:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829417.154121:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829417/real 1787829417] req@ffff9e6803e9a580 x1874668790702080/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829417.154124:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6053 00000100:00080000:0.0:1787829417.154130:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829417.154145:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829417.154148:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790702080/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829417.154153:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829417.154161:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829417.154167:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790702720/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829417.154168:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829417.154176:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790702080:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829417.154184:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829417.154190:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829417.154192:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829417.154198:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6053 00000100:00080000:0.0:1787829417.154203:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829417.154208:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829417.154211:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790702208:192.168.203.109@tcp:38:kworker.0 00000008:00100000:0.0:1787829417.154217:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829417.154236:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790702848/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829417.154237:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829417.154250:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829417.154252:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829417/real 1787829417] req@ffff9e6803e99680 x1874668790702208/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829417.154256:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829417.154260:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829417.154269:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6053 00000100:00080000:0.0:1787829417.154272:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829417.154273:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790702208/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829417.154279:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829417.154288:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787829417.154289:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829417.154295:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829417.154298:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790702208:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829417.154306:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790702336:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829417.154309:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790702976/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829417.154319:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829417.154323:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1787829417.154324:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829417.154327:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829417.154333:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6053 00000100:00080000:0.0:1787829417.154338:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829417.154338:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829417/real 1787829417] req@ffff9e6803e99e00 x1874668790702336/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829417.154344:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829417.154353:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829417.154357:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790702336/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829417.154365:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829417.154366:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790703104/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829417.154370:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829417.154378:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790702336:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829417.154388:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790702464:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829417.154404:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829417.154411:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829417/real 1787829417] req@ffff9e6803e9b480 x1874668790702464/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829417.154419:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790702464/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829417.154425:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829417.154426:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829417.154428:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790702464:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829417.154432:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790702592:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829417.154442:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829417.154447:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829417/real 1787829417] req@ffff9e6803e9b840 x1874668790702592/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829417.154454:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790702592/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829417.154460:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829417.154461:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829417.154463:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790702592:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829417.154467:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790702720:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829417.154476:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829417.154482:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829417/real 1787829417] req@ffff9e6803e9a1c0 x1874668790702720/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829417.154490:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790702720/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829417.154501:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829417.154503:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829417.154505:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790702720:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829417.154518:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790702848:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829417.154528:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829417.154534:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829417/real 1787829417] req@ffff9e6803e98780 x1874668790702848/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829417.154541:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790702848/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829417.154546:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829417.154548:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829417.154550:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790702848:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829417.154553:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790702976:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829417.154562:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829417.154568:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829417/real 1787829417] req@ffff9e6803e98b40 x1874668790702976/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829417.154575:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790702976/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829417.154580:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829417.154582:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829417.154585:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790702976:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829417.154588:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790703104:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829417.154598:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829417.154604:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829417/real 1787829417] req@ffff9e6803e9a940 x1874668790703104/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829417.154610:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790703104/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829417.154616:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829417.154618:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829417.154619:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790703104:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829417.665834:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829417.857889:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829418.689795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829418.881860:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829419.713823:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829419.905926:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829420.737815:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829420.929875:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829421.761804:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829421.953881:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829422.273801:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829422.273809:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829422.273813:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829422.273822:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6058 00000100:00080000:0.0:1787829422.273831:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829422.273838:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829422.273868:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790703232/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829422.273893:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829422.273895:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829422.273897:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829422.273899:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6058 00000100:00080000:0.0:1787829422.273901:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829422.273904:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829422.273911:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829422.273920:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790703360/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829422.273927:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829422.273930:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829422.273931:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829422.273937:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6058 00000100:00100000:3.0:1787829422.273939:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790703232:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829422.273940:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829422.273946:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829422.273952:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829422.273959:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790703488/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829422.273966:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829422.273968:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829422.273969:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829422.273971:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6058 00000100:00080000:0.0:1787829422.273974:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829422.273977:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829422.273980:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829422.273997:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790703616/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829422.274003:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829422.274006:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829422.274007:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829422.274010:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6058 00000100:00080000:0.0:1787829422.274012:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829422.274014:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829422.274018:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787829422.274020:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829422.274029:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790703744/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829422.274035:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829422.274038:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829422.274039:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829422.274042:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6058 00000100:00080000:0.0:1787829422.274045:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829422.274048:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829422.274051:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829422.274058:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790703872/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829422.274068:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829422.274070:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829422.274071:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829422.274073:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6058 00000100:00080000:0.0:1787829422.274076:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829422.274082:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829422.274082:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829422/real 1787829422] req@ffff9e6803e9bc00 x1874668790703232/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829422.274087:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829422.274101:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790704000/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829422.274109:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790703232/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829422.274116:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829422.274126:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829422.274130:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829422.274131:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829422.274137:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6058 00000100:00080000:3.0:1787829422.274140:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829422.274145:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829422.274147:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790703232:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829422.274154:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829422.274160:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829422.274174:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790704128/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829422.274178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790703360:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829422.274195:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829422.274200:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829422.274203:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829422.274208:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6058 00000800:00000100:3.0:1787829422.274212:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829422.274217:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829422.274224:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829422.274230:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829422/real 1787829422] req@ffff9e6803e992c0 x1874668790703360/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829422.274239:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829422.274247:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790703360/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829422.274256:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790704256/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829422.274260:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829422.274268:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829422.274273:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790703360:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829422.274282:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790703488:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829422.274311:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829422.274323:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829422/real 1787829422] req@ffff9e6803e98f00 x1874668790703488/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829422.274335:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790703488/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829422.274344:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829422.274347:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829422.274351:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790703488:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829422.274357:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790703616:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829422.274377:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829422.274388:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829422/real 1787829422] req@ffff9e6803e9ad00 x1874668790703616/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829422.274408:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790703616/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829422.274416:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829422.274420:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829422.274423:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790703616:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829422.274430:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790703744:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829422.274446:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829422.274456:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829422/real 1787829422] req@ffff9e6803e9a940 x1874668790703744/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829422.274478:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790703744/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829422.274487:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829422.274490:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829422.274493:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790703744:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829422.274499:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790703872:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829422.274516:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829422.274526:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829422/real 1787829422] req@ffff9e6803e98b40 x1874668790703872/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829422.274538:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790703872/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829422.274555:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829422.274558:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829422.274562:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790703872:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829422.274568:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790704000:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829422.274586:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829422.274598:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829422/real 1787829422] req@ffff9e6803e98780 x1874668790704000/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829422.274612:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790704000/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829422.274621:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829422.274624:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829422.274628:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790704000:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829422.274648:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790704128:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829422.274667:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829422.274680:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829422/real 1787829422] req@ffff9e6803e9a1c0 x1874668790704128/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829422.274692:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790704128/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829422.274734:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829422.274739:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829422.274743:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790704128:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829422.274749:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790704256:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829422.274769:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829422.274779:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829422/real 1787829422] req@ffff9e6803e9b840 x1874668790704256/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829422.274791:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790704256/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829422.274799:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829422.274802:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829422.274805:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790704256:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829422.785840:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829422.977832:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829423.809844:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829424.001942:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829424.833845:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829425.025906:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829425.857837:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829426.049899:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829426.881810:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829427.073854:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829427.393823:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829427.393837:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829427.393843:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829427.393857:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6064 00000100:00080000:0.0:1787829427.393872:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829427.393882:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829427.393940:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790704384/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829427.393968:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829427.393971:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829427.393973:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829427.393977:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6064 00000100:00080000:0.0:1787829427.393983:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829427.393988:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829427.394002:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829427.394009:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790704384:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787829427.394026:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790704512/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829427.394042:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829427.394050:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829427.394052:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829427.394057:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6064 00000100:00080000:0.0:1787829427.394063:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829427.394069:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829427.394075:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829427.394090:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790704640/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829427.394102:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787829427.394110:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829427.394112:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829427.394117:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829427.394125:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6064 00000100:00080000:0.0:1787829427.394140:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829427.394153:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829427.394159:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829427/real 1787829427] req@ffff9e6803e9b480 x1874668790704384/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829427.394164:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829427.394187:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790704768/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829427.394197:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790704384/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829427.394202:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829427.394215:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829427.394221:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829427.394229:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829427.394238:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829427.394239:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6064 00000100:00080000:0.0:1787829427.394249:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829427.394250:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790704384:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829427.394261:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829427.394271:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829427.394291:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790704896/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829427.394292:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790704512:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829427.394316:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829427.394321:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829427.394324:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787829427.394326:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829427.394335:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6064 00000100:00080000:0.0:1787829427.394341:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829427.394349:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829427.394356:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829427.394361:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829427/real 1787829427] req@ffff9e6803e99e00 x1874668790704512/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829427.394378:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790705024/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829427.394382:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790704512/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829427.394409:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829427.394416:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829427.394424:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790704512:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829427.394427:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829427.394433:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790704640:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829427.394436:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829427.394440:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829427.394447:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6064 00000100:00080000:0.0:1787829427.394453:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829427.394466:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:3.0:1787829427.394467:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787829427.394484:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829427.394490:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829427/real 1787829427] req@ffff9e6803e99680 x1874668790704640/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829427.394500:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790705152/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829427.394524:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829427.394527:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829427.394531:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829427.394533:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790704640/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829427.394538:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6064 00000100:00080000:0.0:1787829427.394545:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829427.394553:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829427.394558:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1787829427.394567:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829427.394570:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790705280/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829427.394583:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829427.394593:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829427.394595:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790704640:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829427.394604:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829427.394607:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829427.394613:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790704768:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787829427.394614:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6064 00000100:00080000:0.0:1787829427.394625:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829427.394632:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829427.394638:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787829427.394648:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829427.394656:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790705408/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829427.394667:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829427/real 1787829427] req@ffff9e6803e9a580 x1874668790704768/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829427.394727:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790704768/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829427.394738:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829427.394740:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829427.394743:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790704768:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829427.394760:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790704896:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829427.394778:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829427.394789:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829427/real 1787829427] req@ffff9e6803e9b840 x1874668790704896/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829427.394801:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790704896/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829427.394810:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829427.394814:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829427.394817:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790704896:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829427.394824:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790705024:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829427.394845:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829427.394853:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829427/real 1787829427] req@ffff9e6803e9a1c0 x1874668790705024/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829427.394861:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790705024/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829427.394872:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829427.394874:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829427.394876:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790705024:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829427.394880:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790705152:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829427.394891:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829427.394897:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829427/real 1787829427] req@ffff9e6803e98780 x1874668790705152/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829427.394904:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790705152/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829427.394909:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829427.394911:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829427.394913:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790705152:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829427.394916:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790705280:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829427.394926:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829427.394932:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829427/real 1787829427] req@ffff9e6803e98b40 x1874668790705280/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829427.394939:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790705280/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829427.394944:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829427.394946:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829427.394948:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790705280:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829427.394952:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790705408:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829427.394963:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829427.394968:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829427/real 1787829427] req@ffff9e6803e9a940 x1874668790705408/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829427.394975:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790705408/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829427.394980:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829427.394982:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829427.394984:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790705408:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829427.905815:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829428.097870:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829428.929770:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829429.121859:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829429.953806:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829430.145837:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829430.977849:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829431.169905:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829432.001788:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829432.193828:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829432.513807:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829432.513819:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829432.513823:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829432.513833:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6069 00000100:00080000:1.0:1787829432.513844:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829432.513853:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829432.513891:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790705536/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829432.513916:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829432.513921:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829432.513923:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829432.513930:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6069 00000100:00080000:1.0:1787829432.513936:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829432.513941:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829432.513946:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790705536:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787829432.513954:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829432.513975:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790705664/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829432.513995:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829432.513999:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829432.514000:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829432.514004:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6069 00000100:00080000:1.0:1787829432.514007:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829432.514010:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829432.514018:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829432.514029:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790705792/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829432.514036:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790705664:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829432.514040:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829432.514044:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829432.514046:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829432.514052:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6069 00000100:00080000:1.0:1787829432.514058:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829432.514065:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790705792:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829432.514066:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829432.514077:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829432.514095:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790705920/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829432.514109:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829432.514112:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829432.514114:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829432.514117:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6069 00000100:00080000:1.0:1787829432.514122:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829432.514126:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790705920:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787829432.514127:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829432.514137:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829432.514149:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790706048/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829432.514157:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829432.514159:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829432.514160:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829432.514164:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6069 00000100:00080000:1.0:1787829432.514167:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829432.514169:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790706048:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787829432.514173:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829432.514179:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829432.514200:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790706176/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829432.514216:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829432.514221:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829432.514223:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829432.514228:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6069 00000100:00080000:1.0:1787829432.514234:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829432.514237:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790706176:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829432.514243:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829432.514251:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829432.514267:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790706304/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829432.514284:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829432.514288:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829432.514290:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829432.514295:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6069 00000100:00100000:3.0:1787829432.514297:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790706304:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829432.514302:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829432.514308:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829432.514313:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829432.514330:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790706432/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829432.514346:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829432.514350:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829432.514352:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829432.514357:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790706432:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787829432.514359:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6069 00000100:00080000:1.0:1787829432.514367:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829432.514372:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829432.514379:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829432.514393:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790706560/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829432.514449:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790706560:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829433.025809:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829433.217846:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829434.049796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829434.241822:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829435.073798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829435.265860:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787829435.585917:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:2.0:1787829435.585965:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787829435.585978:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787829435.586015:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:2.0:1787829435.586024:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787829435.586032:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787829435.586039:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787829435.586045:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:3.0:1787829435.586047:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829432/real 1787829435] req@ffff9e6831aa8b40 x1874668790706560/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787829435.586054:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787829435.586067:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787829435.586083:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:3.0:1787829435.586086:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790706560/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829435.586099:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829435.586105:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829435.586110:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790706560:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829435.586120:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829432/real 1787829435] req@ffff9e6831aa9e00 x1874668790706432/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829435.586133:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790706432/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829435.586140:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829435.586151:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829435.586154:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790706432:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829435.586161:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829432/real 1787829435] req@ffff9e6831aa8f00 x1874668790706304/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829435.586174:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790706304/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829435.586188:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829435.586192:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829435.586195:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790706304:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829435.586202:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829432/real 1787829435] req@ffff9e6831aaa580 x1874668790706176/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829435.586213:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790706176/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829435.586222:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829435.586226:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829435.586229:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790706176:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829435.586235:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829432/real 1787829435] req@ffff9e6831aa9680 x1874668790706048/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829435.586247:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790706048/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829435.586254:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829435.586257:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829435.586260:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790706048:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829435.586265:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829432/real 1787829435] req@ffff9e6831aa9a40 x1874668790705920/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829435.586275:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790705920/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829435.586283:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829435.586286:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829435.586289:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790705920:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829435.586294:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829432/real 1787829435] req@ffff9e6831aabc00 x1874668790705792/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829435.586305:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790705792/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829435.586313:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829435.586316:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829435.586319:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790705792:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829435.586325:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829432/real 1787829435] req@ffff9e6831aaa940 x1874668790705664/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829435.586336:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790705664/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829435.586360:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829435.586364:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829435.586368:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790705664:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829435.586374:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829432/real 1787829435] req@ffff9e6831aa8000 x1874668790705536/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829435.586385:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790705536/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829435.586394:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829435.586397:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829435.586401:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790705536:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829436.097791:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829436.289856:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829437.121820:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829437.313852:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829437.633802:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829437.633815:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829437.633821:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829437.633837:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6074 00000100:00080000:1.0:1787829437.633852:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829437.633862:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829437.633902:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790706688/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829437.633930:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829437.633933:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829437.633935:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829437.633938:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6074 00000100:00080000:1.0:1787829437.633941:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829437.633944:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829437.633953:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829437.633964:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790706816/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829437.633972:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829437.633975:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829437.633977:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829437.633982:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6074 00000100:00080000:1.0:1787829437.633988:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829437.633988:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790706688:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829437.633996:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829437.634004:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829437.634032:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790706944/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829437.634042:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829437.634046:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829437.634049:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829437.634053:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6074 00000100:00080000:1.0:1787829437.634058:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829437.634066:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829437.634072:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829437.634081:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829437.634098:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790707072/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829437.634110:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829437.634114:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829437.634117:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829437.634123:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829437/real 1787829437] req@ffff9e6831aaad00 x1874668790706688/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829437.634128:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6074 00000100:00100000:3.0:1787829437.634147:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790706688/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829437.634156:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829437.634162:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829437.634167:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:3.0:1787829437.634172:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:1.0:1787829437.634183:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829437.634184:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790706688:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787829437.634209:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790707200/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829437.634226:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829437.634228:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790706816:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829437.634238:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829437.634244:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829437.634253:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6074 00000100:00080000:1.0:1787829437.634260:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829437.634266:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:3.0:1787829437.634280:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1787829437.634282:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829437.634304:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790707328/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829437.634315:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829437/real 1787829437] req@ffff9e6831aaa1c0 x1874668790706816/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829437.634324:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829437.634338:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829437.634342:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829437.634354:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6074 00000100:00080000:1.0:1787829437.634360:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829437.634363:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790706816/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829437.634370:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829437.634382:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829437.634388:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829437.634399:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829437.634402:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790707456/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829437.634412:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790706816:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829437.634425:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790706944:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829437.634429:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829437.634437:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829437.634440:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829437.634447:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6074 00000100:00080000:1.0:1787829437.634455:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829437.634461:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829437.634468:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787829437.634476:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829437.634488:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790707584/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829437.634501:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829437/real 1787829437] req@ffff9e6831aa8780 x1874668790706944/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829437.634505:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829437.634520:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829437.634527:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829437.634540:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6074 00000100:00080000:1.0:1787829437.634546:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829437.634554:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829437.634556:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790706944/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829437.634563:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829437.634582:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790707712/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829437.634847:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829437.634853:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829437.634858:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790706944:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829437.634868:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790707072:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829437.634899:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829437.634913:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829437/real 1787829437] req@ffff9e6831aab480 x1874668790707072/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829437.634926:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790707072/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829437.634934:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829437.634938:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829437.634942:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790707072:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829437.634971:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790707200:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829437.634994:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829437.635007:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829437/real 1787829437] req@ffff9e6831aa8000 x1874668790707200/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829437.635019:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790707200/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829437.635028:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829437.635031:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829437.635035:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790707200:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829437.635042:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790707328:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829437.635063:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829437.635076:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829437/real 1787829437] req@ffff9e6831aaa940 x1874668790707328/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829437.635090:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790707328/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829437.635101:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829437.635103:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829437.635106:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790707328:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829437.635110:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790707456:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829437.635126:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829437.635134:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829437/real 1787829437] req@ffff9e6831aabc00 x1874668790707456/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829437.635141:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790707456/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829437.635147:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829437.635149:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829437.635151:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790707456:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829437.635156:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790707584:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829437.635168:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829437.635177:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829437/real 1787829437] req@ffff9e6831aa9a40 x1874668790707584/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829437.635189:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790707584/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829437.635196:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829437.635200:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829437.635203:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790707584:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829437.635209:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790707712:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829437.635229:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829437.635241:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829437/real 1787829437] req@ffff9e6831aa9680 x1874668790707712/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829437.635251:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790707712/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829437.635260:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829437.635263:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829437.635266:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790707712:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829438.145851:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829438.337900:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829439.169818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829439.361905:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829440.193776:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829440.385864:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829441.217818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829441.409886:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829442.241799:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829442.433836:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829442.753817:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829442.753826:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829442.753834:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829442.753843:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6079 00000100:00080000:1.0:1787829442.753853:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829442.753860:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829442.753900:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790707840/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829442.753921:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829442.753924:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829442.753925:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829442.753930:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6079 00000100:00080000:1.0:1787829442.753932:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829442.753935:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829442.753944:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829442.753953:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790707968/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829442.753962:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829442.753965:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829442.753966:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829442.753972:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6079 00000100:00080000:1.0:1787829442.753978:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829442.753983:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829442.753990:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790707840:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787829442.753992:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829442.754022:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790708096/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829442.754030:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829442.754033:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829442.754034:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829442.754037:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6079 00000100:00080000:1.0:1787829442.754040:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829442.754043:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829442.754047:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829442.754054:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790708224/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829442.754062:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829442.754065:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829442.754066:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829442.754068:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6079 00000100:00080000:1.0:1787829442.754080:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829442.754083:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829442.754092:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1787829442.754099:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829442.754101:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790708352/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829442.754111:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829442.754114:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829442.754116:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829442.754118:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6079 00000100:00080000:1.0:1787829442.754124:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829442.754129:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829442.754135:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829442.754149:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790708480/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829442.754167:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829442.754170:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829442/real 1787829442] req@ffff9e6831aaa580 x1874668790707840/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829442.754173:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829442.754177:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829442.754188:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6079 00000100:00080000:1.0:1787829442.754198:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829442.754204:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829442.754209:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829442.754216:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790707840/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829442.754221:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790708608/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829442.754269:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829442.754270:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829442.754272:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829442.754276:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829442.754284:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6079 00000100:00080000:3.0:1787829442.754286:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829442.754288:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829442.754297:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829442.754303:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790707840:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787829442.754306:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829442.754328:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790708736/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829442.754335:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829442.754338:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829442.754339:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829442.754341:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6079 00000100:00080000:1.0:1787829442.754347:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829442.754350:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829442.754353:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829442.754355:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790707968:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829442.754363:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790708864/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829442.754393:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829442.754413:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829442/real 1787829442] req@ffff9e6831aa8f00 x1874668790707968/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829442.754428:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790707968/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829442.754438:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829442.754442:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829442.754445:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790707968:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829442.754453:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790708096:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829442.754471:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829442.754481:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829442/real 1787829442] req@ffff9e6831aa9e00 x1874668790708096/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829442.754493:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790708096/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829442.754502:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829442.754504:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829442.754506:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790708096:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829442.754511:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790708224:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829442.754528:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829442.754535:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829442/real 1787829442] req@ffff9e6831aa8b40 x1874668790708224/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829442.754543:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790708224/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829442.754548:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829442.754550:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829442.754552:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790708224:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829442.754556:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790708352:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829442.754567:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829442.754593:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829442/real 1787829442] req@ffff9e6831aa9680 x1874668790708352/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829442.754614:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790708352/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829442.754622:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829442.754625:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829442.754629:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790708352:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829442.754635:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790708480:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829442.754658:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829442.754670:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829442/real 1787829442] req@ffff9e6831aa9a40 x1874668790708480/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829442.754683:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790708480/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829442.754718:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829442.754723:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829442.754727:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790708480:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829442.754734:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790708608:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829442.754759:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829442.754772:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829442/real 1787829442] req@ffff9e6831aabc00 x1874668790708608/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829442.754785:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790708608/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829442.754800:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829442.754803:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829442.754807:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790708608:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829442.754814:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790708736:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829442.754836:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829442.754847:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829442/real 1787829442] req@ffff9e6831aaa940 x1874668790708736/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829442.754860:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790708736/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829442.754869:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829442.754872:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829442.754876:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790708736:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829442.754900:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790708864:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829442.754918:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829442.754928:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829442/real 1787829442] req@ffff9e6831aa8000 x1874668790708864/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829442.754940:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790708864/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829442.754948:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829442.754950:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829442.754953:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790708864:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829443.265849:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829443.457902:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829444.289827:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829444.481914:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829445.313849:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829445.505932:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829446.337862:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829446.529919:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829447.361840:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829447.553924:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829447.873834:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829447.873845:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829447.873849:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829447.873859:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6084 00000100:00080000:0.0:1787829447.873870:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829447.873876:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829447.873909:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790708992/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829447.873939:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829447.873944:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829447.873946:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829447.873951:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6084 00000100:00080000:0.0:1787829447.873957:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829447.873963:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829447.873974:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829447.873991:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790709120/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829447.873993:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790708992:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829447.874003:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829447.874007:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829447.874009:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829447.874012:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6084 00000100:00080000:0.0:1787829447.874016:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829447.874019:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829447.874024:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829447.874040:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790709248/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829447.874051:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829447.874054:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829447.874056:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787829447.874058:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829447.874060:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6084 00000100:00080000:0.0:1787829447.874064:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829447.874069:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829447.874076:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829447.874092:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790709376/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829447.874103:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829447.874107:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829447.874109:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829447.874110:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829447/real 1787829447] req@ffff9e6803e9ad00 x1874668790708992/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829447.874114:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6084 00000100:00080000:0.0:1787829447.874118:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829447.874124:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829447.874129:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829447.874136:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790708992/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829447.874146:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790709504/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829447.874151:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829447.874158:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829447.874160:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829447.874164:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1787829447.874166:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790708992:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829447.874167:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829447.874173:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6084 00000100:00080000:0.0:1787829447.874178:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829447.874184:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829447.874190:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829447.874204:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790709632/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829447.874220:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829447.874223:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829447.874226:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829447.874232:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6084 00000100:00080000:0.0:1787829447.874236:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829447.874242:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829447.874248:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829447.874261:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790709760/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829447.874270:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829447.874273:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829447.874275:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829447.874280:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6084 00000100:00080000:0.0:1787829447.874284:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829447.874289:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829447.874294:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829447.874309:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790709888/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829447.874319:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829447.874322:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829447.874325:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829447.874329:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6084 00000100:00080000:0.0:1787829447.874334:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829447.874338:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829447.874344:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829447.874353:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790710016/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829447.874424:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790709120:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829447.874494:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829447.874514:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829447/real 1787829447] req@ffff9e6803e98f00 x1874668790709120/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829447.874528:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790709120/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829447.874538:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829447.874555:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829447.874559:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790709120:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829447.874567:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790709248:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829447.874585:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829447.874593:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829447/real 1787829447] req@ffff9e6803e992c0 x1874668790709248/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829447.874601:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790709248/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829447.874606:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829447.874608:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829447.874610:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790709248:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829447.874614:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790709376:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829447.874624:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829447.874630:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829447/real 1787829447] req@ffff9e6803e9bc00 x1874668790709376/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829447.874637:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790709376/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829447.874642:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829447.874644:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829447.874646:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790709376:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829447.874649:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790709504:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829447.874659:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829447.874664:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829447/real 1787829447] req@ffff9e6803e9a940 x1874668790709504/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829447.874672:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790709504/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829447.874677:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829447.874678:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829447.874680:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790709504:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829447.874683:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790709632:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829447.874693:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829447.874734:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829447/real 1787829447] req@ffff9e6803e98b40 x1874668790709632/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829447.874748:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790709632/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829447.874758:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829447.874762:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829447.874765:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790709632:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829447.874772:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790709760:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829447.874791:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829447.874802:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829447/real 1787829447] req@ffff9e6803e98780 x1874668790709760/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829447.874809:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790709760/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829447.874819:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829447.874821:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829447.874823:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790709760:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829447.874827:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790709888:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829447.874838:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829447.874844:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829447/real 1787829447] req@ffff9e6803e9a1c0 x1874668790709888/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829447.874850:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790709888/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829447.874855:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829447.874857:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829447.874859:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790709888:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829447.874862:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790710016:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829447.874872:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829447.874877:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829447/real 1787829447] req@ffff9e6803e9b840 x1874668790710016/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829447.874883:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790710016/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829472 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829447.874888:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829447.874890:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829447.874892:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790710016:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829448.385790:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829448.577943:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829449.409856:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829449.601877:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829450.433809:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829450.625895:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829451.457810:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829451.649854:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829452.481799:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829452.673889:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829452.993803:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829452.993816:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829452.993820:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829452.993831:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6089 00000100:00080000:0.0:1787829452.993844:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829452.993851:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829452.993905:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790710144/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829452.993937:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829452.993943:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829452.993945:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829452.993950:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6089 00000100:00080000:0.0:1787829452.993956:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829452.993961:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829452.993973:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790710144:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1787829452.993974:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829452.993994:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790710272/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829452.994005:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829452.994010:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829452.994012:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829452.994016:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6089 00000100:00080000:0.0:1787829452.994021:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829452.994025:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829452.994031:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829452.994045:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790710400/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829452.994058:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829452.994059:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829452.994065:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829452.994067:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829452.994072:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6089 00000100:00080000:0.0:1787829452.994076:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829452.994080:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829452.994085:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829452.994096:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790710528/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829452.994104:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829452.994107:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829452/real 1787829452] req@ffff9e6803e9a580 x1874668790710144/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829452.994108:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829452.994111:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829452.994119:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6089 00000100:00080000:0.0:1787829452.994125:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829452.994128:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829452.994132:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829452.994135:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790710144/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829452.994144:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790710656/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829452.994146:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829452.994156:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829452.994157:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829452.994162:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1787829452.994163:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790710144:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829452.994164:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829452.994172:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6089 00000100:00080000:0.0:1787829452.994175:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829452.994177:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829452.994181:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829452.994188:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790710784/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829452.994195:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790710272:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829452.994196:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829452.994202:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829452.994203:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829452.994207:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6089 00000100:00080000:0.0:1787829452.994210:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829452.994213:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829452.994216:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829452.994220:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829452.994228:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790710912/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829452.994235:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829452/real 1787829452] req@ffff9e6803e99680 x1874668790710272/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829452.994238:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829452.994251:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790710272/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829452.994255:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829452.994259:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829452.994264:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829452.994265:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6089 00000100:00080000:0.0:1787829452.994270:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829452.994275:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829452.994278:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829452.994284:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829452.994285:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790710272:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829452.994295:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790711040/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829452.994299:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790710400:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829452.994315:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829452.994320:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829452.994322:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787829452.994327:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829452.994330:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6089 00000100:00080000:0.0:1787829452.994337:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829452.994343:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829452.994348:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829452/real 1787829452] req@ffff9e6803e99e00 x1874668790710400/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829452.994351:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829452.994367:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790711168/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829452.994368:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790710400/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829452.994394:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829452.994397:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829452.994400:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790710400:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829452.994406:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790710528:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829452.994424:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829452.994431:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829452/real 1787829452] req@ffff9e6803e9b480 x1874668790710528/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829452.994439:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790710528/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829452.994444:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829452.994446:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829452.994448:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790710528:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829452.994452:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790710656:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829452.994462:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829452.994468:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829452/real 1787829452] req@ffff9e6803e9b840 x1874668790710656/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829452.994480:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790710656/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829452.994486:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829452.994487:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829452.994489:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790710656:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829452.994501:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790710784:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829452.994532:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829452.994541:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829452/real 1787829452] req@ffff9e6803e9a1c0 x1874668790710784/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829452.994548:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790710784/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829452.994555:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829452.994557:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829452.994559:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790710784:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829452.994563:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790710912:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829452.994574:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829452.994580:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829452/real 1787829452] req@ffff9e6803e98780 x1874668790710912/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829452.994588:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790710912/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829452.994598:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829452.994600:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829452.994602:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790710912:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829452.994605:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790711040:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829452.994614:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829452.994620:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829452/real 1787829452] req@ffff9e6803e98b40 x1874668790711040/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829452.994627:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790711040/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829452.994633:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829452.994635:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829452.994636:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790711040:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829452.994640:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790711168:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829452.994649:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829452.994655:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829452/real 1787829452] req@ffff9e6803e9a940 x1874668790711168/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829452.994661:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790711168/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829477 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829452.994666:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829452.994668:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829452.994671:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790711168:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829453.505829:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829453.697846:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829454.529834:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829454.721889:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829455.553767:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829455.746626:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829456.577833:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829456.769840:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829457.601844:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829457.793828:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829458.113813:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829458.113822:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829458.113826:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829458.113834:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6094 00000100:00080000:3.0:1787829458.113844:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829458.113850:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829458.113881:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790711296/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829458.113906:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829458.113909:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829458.113910:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829458.113913:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6094 00000100:00080000:3.0:1787829458.113916:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829458.113918:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829458.113930:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829458.113940:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790711424/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829458.113947:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829458.113950:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829458.113951:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829458.113954:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6094 00000100:00080000:3.0:1787829458.113957:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829458.113961:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829458.113966:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790711296:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787829458.113966:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829458.113975:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790711552/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829458.113983:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829458.113986:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829458.113987:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829458.113990:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6094 00000100:00080000:3.0:1787829458.113992:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829458.113994:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829458.113997:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829458.114005:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790711680/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829458.114011:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829458.114014:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829458.114015:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829458.114017:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6094 00000100:00080000:3.0:1787829458.114019:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829458.114022:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829458.114025:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829458.114034:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790711808/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829458.114035:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829458.114045:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829458.114048:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829458.114050:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829458.114053:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6094 00000100:00080000:3.0:1787829458.114056:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829458.114058:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829458.114061:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829458.114071:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790711936/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829458.114077:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829458/real 1787829458] req@ffff9e680a2d4000 x1874668790711296/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829458.114081:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829458.114089:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829458.114092:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829458.114099:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6094 00000100:00080000:3.0:1787829458.114102:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829458.114106:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790711296/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829458.114108:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829458.114115:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829458.114124:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829458.114129:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790712064/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829458.114131:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829458.114138:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790711296:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829458.114149:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829458.114152:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790711424:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829458.114154:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829458.114158:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829458.114162:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6094 00000100:00080000:3.0:1787829458.114165:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829458.114168:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829458.114171:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829458.114177:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829458.114182:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790712192/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829458.114190:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829458/real 1787829458] req@ffff9e680a2d6580 x1874668790711424/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829458.114191:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829458.114199:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829458.114202:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829458.114210:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6094 00000100:00100000:1.0:1787829458.114213:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790711424/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829458.114215:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829458.114222:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829458.114229:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829458.114231:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829458.114236:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829458.114240:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790712320/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829458.114243:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790711424:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829458.114275:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790711552:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829458.114299:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829458.114311:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829458/real 1787829458] req@ffff9e680a2d70c0 x1874668790711552/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829458.114324:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790711552/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829458.114330:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829458.114332:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829458.114334:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790711552:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829458.114339:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790711680:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829458.114353:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829458.114360:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829458/real 1787829458] req@ffff9e680a2d43c0 x1874668790711680/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829458.114368:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790711680/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829458.114373:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829458.114375:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829458.114377:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790711680:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829458.114380:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790711808:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829458.114390:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829458.114396:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829458/real 1787829458] req@ffff9e680a2d4780 x1874668790711808/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829458.114403:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790711808/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829458.114409:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829458.114410:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829458.114412:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790711808:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829458.114416:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790711936:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829458.114425:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829458.114431:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829458/real 1787829458] req@ffff9e680a2d52c0 x1874668790711936/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829458.114438:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790711936/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829458.114444:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829458.114446:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829458.114448:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790711936:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829458.114451:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790712064:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829458.114467:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829458.114492:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829458/real 1787829458] req@ffff9e680a2d61c0 x1874668790712064/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829458.114513:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790712064/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829458.114522:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829458.114524:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829458.114526:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790712064:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829458.114531:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790712192:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829458.114546:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829458.114554:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829458/real 1787829458] req@ffff9e680a2d6d00 x1874668790712192/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829458.114561:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790712192/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829458.114566:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829458.114568:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829458.114570:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790712192:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829458.114573:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790712320:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829458.114583:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829458.114588:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829458/real 1787829458] req@ffff9e680a2d5e00 x1874668790712320/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829458.114595:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790712320/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829458.114604:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829458.114607:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829458.114609:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790712320:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829458.625819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829458.817893:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829459.649828:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829459.841844:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829460.673823:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829460.865924:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829461.697818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829461.889885:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829462.721854:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829462.913908:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829463.233813:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829463.233824:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829463.233828:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829463.233836:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6099 00000100:00080000:2.0:1787829463.233847:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829463.233853:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829463.233885:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790712448/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829463.233908:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829463.233911:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829463.233912:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829463.233915:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6099 00000100:00080000:2.0:1787829463.233917:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829463.233919:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829463.233926:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829463.233941:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790712576/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829463.233948:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829463.233951:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829463.233952:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829463.233955:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6099 00000100:00080000:2.0:1787829463.233959:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829463.233961:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829463.233965:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829463.233972:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790712704/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829463.233980:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829463.233983:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790712448:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829463.233994:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829463.233997:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829463.234001:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6099 00000100:00080000:2.0:1787829463.234006:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829463.234011:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829463.234017:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829463.234032:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790712832/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829463.234041:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829463.234046:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829463.234047:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829463.234051:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829463.234055:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6099 00000100:00080000:2.0:1787829463.234060:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829463.234063:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829463.234067:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829463.234078:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790712960/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829463.234085:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829463/real 1787829463] req@ffff9e6804420f00 x1874668790712448/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829463.234094:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829463.234098:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829463.234099:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829463.234104:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790712448/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829463.234106:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6099 00000100:00080000:2.0:1787829463.234115:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829463.234119:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829463.234124:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787829463.234126:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829463.234132:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790712448:192.168.203.109@tcp:250:kworker.0 00000008:00100000:2.0:1787829463.234135:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829463.234140:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790712576:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829463.234155:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790713088/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829463.234162:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829463.234175:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829463/real 1787829463] req@ffff9e6804422580 x1874668790712576/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829463.234184:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829463.234185:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790712576/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829463.234195:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829463.234201:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829463.234210:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829463.234212:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6099 00000100:00080000:1.0:1787829463.234216:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829463.234219:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829463.234220:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790712576:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829463.234228:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829463.234233:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829463.234245:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790712704:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829463.234249:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790713216/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829463.234262:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829463.234265:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829463.234272:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829463/real 1787829463] req@ffff9e6804421680 x1874668790712704/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829463.234272:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829463.234277:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829463.234288:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6099 00000100:00080000:2.0:1787829463.234294:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829463.234296:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790712704/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829463.234303:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787829463.234314:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787829463.234315:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829463.234323:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829463.234327:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790712704:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829463.234334:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790713344/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829463.234337:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790712832:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787829463.234354:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829463.234357:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829463.234359:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829463.234363:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6099 00000800:00000100:1.0:1787829463.234368:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829463.234370:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829463.234376:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829463.234381:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829463.234382:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829463/real 1787829463] req@ffff9e6804423480 x1874668790712832/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829463.234395:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790713472/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829463.234397:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790712832/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829463.234428:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829463.234432:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829463.234539:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790712832:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829463.234545:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790712960:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829463.234560:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829463.234568:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829463/real 1787829463] req@ffff9e6804422940 x1874668790712960/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829463.234576:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790712960/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829463.234582:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829463.234584:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829463.234586:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790712960:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829463.234589:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790713088:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829463.234599:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829463.234605:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829463/real 1787829463] req@ffff9e6804422d00 x1874668790713088/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829463.234612:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790713088/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829463.234618:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829463.234620:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829463.234622:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790713088:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829463.234632:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790713216:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829463.234642:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829463.234648:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829463/real 1787829463] req@ffff9e6804420780 x1874668790713216/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829463.234655:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790713216/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829463.234660:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829463.234662:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829463.234663:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790713216:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829463.234667:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790713344:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829463.234676:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829463.234681:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829463/real 1787829463] req@ffff9e68044221c0 x1874668790713344/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829463.234688:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790713344/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829463.234694:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829463.234713:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829463.234715:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790713344:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829463.234719:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790713472:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829463.234731:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829463.234737:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829463/real 1787829463] req@ffff9e6804423840 x1874668790713472/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829463.234745:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790713472/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829463.234750:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829463.234752:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829463.234754:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790713472:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829463.745782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829463.937893:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829464.769831:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829464.961916:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829465.793766:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829465.985944:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829466.817789:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829467.009885:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829467.841785:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829468.033869:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829468.353840:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829468.353852:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829468.353857:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829468.353868:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6105 00000100:00080000:0.0:1787829468.353880:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829468.353891:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829468.353932:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790713600/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829468.353960:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829468.353963:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829468.353965:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829468.353968:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6105 00000100:00080000:0.0:1787829468.353970:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829468.353972:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829468.353982:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829468.353995:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790713728/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829468.354002:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829468.354005:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829468.354006:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829468.354008:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6105 00000100:00080000:0.0:1787829468.354010:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829468.354012:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829468.354016:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829468.354021:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790713600:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787829468.354026:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790713856/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829468.354036:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829468.354038:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829468.354039:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829468.354041:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6105 00000100:00080000:0.0:1787829468.354044:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829468.354046:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829468.354049:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829468.354058:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790713984/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829468.354063:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829468.354066:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829468.354067:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829468.354069:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6105 00000100:00080000:0.0:1787829468.354071:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829468.354073:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829468.354077:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829468.354084:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790714112/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829468.354090:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829468.354093:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829468.354095:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829468.354097:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6105 00000100:00080000:0.0:1787829468.354099:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829468.354102:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829468.354105:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829468.354110:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829468.354115:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790714240/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829468.354126:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829468.354129:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829468.354130:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829468.354133:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6105 00000100:00080000:0.0:1787829468.354135:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829468.354138:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829468.354140:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829468.354147:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790714368/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829468.354153:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829468.354156:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829468.354157:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829468.354160:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6105 00000100:00080000:0.0:1787829468.354162:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829468.354168:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829468.354169:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829468/real 1787829468] req@ffff9e6803e9bc00 x1874668790713600/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829468.354171:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829468.354182:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790714496/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829468.354190:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829468.354193:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829468.354194:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829468.354198:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790713600/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829468.354200:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6105 00000100:00080000:0.0:1787829468.354205:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829468.354212:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829468.354217:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829468.354222:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829468.354227:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790714624/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829468.354232:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829468.354242:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790713600:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787829468.354270:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790713728:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829468.354294:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829468.354307:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829468/real 1787829468] req@ffff9e6803e992c0 x1874668790713728/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829468.354320:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790713728/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829468.354330:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829468.354334:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829468.354338:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790713728:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829468.354344:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790713856:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829468.354361:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829468.354370:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829468/real 1787829468] req@ffff9e6803e98f00 x1874668790713856/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829468.354383:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790713856/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829468.354391:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829468.354395:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829468.354411:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790713856:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829468.354419:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790713984:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829468.354437:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829468.354448:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829468/real 1787829468] req@ffff9e6803e9ad00 x1874668790713984/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829468.354461:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790713984/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829468.354470:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829468.354474:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829468.354477:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790713984:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829468.354483:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790714112:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829468.354499:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829468.354509:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829468/real 1787829468] req@ffff9e6803e9a940 x1874668790714112/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829468.354521:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790714112/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829468.354530:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829468.354533:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829468.354537:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790714112:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829468.354543:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790714240:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829468.354558:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829468.354568:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829468/real 1787829468] req@ffff9e6803e98b40 x1874668790714240/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829468.354581:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790714240/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829468.354592:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829468.354595:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829468.354599:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790714240:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829468.354613:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790714368:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829468.354630:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829468.354640:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829468/real 1787829468] req@ffff9e6803e98780 x1874668790714368/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829468.354660:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790714368/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829468.354670:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829468.354673:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829468.354676:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790714368:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829468.354682:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790714496:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829468.354726:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829468.354738:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829468/real 1787829468] req@ffff9e6803e9a1c0 x1874668790714496/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829468.354751:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790714496/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829468.354768:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829468.354771:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829468.354774:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790714496:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829468.354780:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790714624:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829468.354801:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829468.354810:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829468/real 1787829468] req@ffff9e6803e9b840 x1874668790714624/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829468.354823:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790714624/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829468.354838:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829468.354841:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829468.354845:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790714624:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829468.865924:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829469.057851:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829469.889822:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829470.081855:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829470.913807:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829471.105883:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829471.937842:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829472.129877:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829472.961830:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829473.153924:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829473.473834:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829473.473857:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829473.473862:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829473.473873:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6110 00000100:00080000:2.0:1787829473.473885:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829473.473893:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829473.473945:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790714752/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829473.473971:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829473.473974:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829473.473976:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829473.473979:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6110 00000100:00080000:2.0:1787829473.473982:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829473.473985:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829473.473996:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829473.474006:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790714880/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829473.474014:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829473.474016:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829473.474018:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829473.474020:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6110 00000100:00080000:2.0:1787829473.474022:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829473.474025:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829473.474030:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790714752:192.168.203.109@tcp:250:kworker.0 00000008:00100000:2.0:1787829473.474030:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829473.474039:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790715008/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829473.474047:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829473.474050:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829473.474051:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829473.474054:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6110 00000100:00080000:2.0:1787829473.474056:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829473.474058:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829473.474061:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829473.474070:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790715136/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829473.474076:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829473.474079:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829473.474080:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829473.474083:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6110 00000100:00080000:2.0:1787829473.474086:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829473.474090:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829473.474095:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829473.474117:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790715264/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829473.474124:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829473.474136:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829473.474141:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829473.474143:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829473.474149:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6110 00000100:00080000:2.0:1787829473.474153:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829473.474159:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829473.474164:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829473.474178:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790715392/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829473.474189:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829473.474191:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829473/real 1787829473] req@ffff9e6804421e00 x1874668790714752/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829473.474192:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829473.474195:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829473.474203:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6110 00000100:00080000:2.0:1787829473.474217:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829473.474220:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829473.474225:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829473.474226:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790714752/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829473.474237:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790715520/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829473.474246:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829473.474247:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829473.474251:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829473.474255:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829473.474259:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829473.474260:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6110 00000100:00080000:2.0:1787829473.474267:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829473.474271:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790714752:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829473.474271:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829473.474277:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829473.474287:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790715648/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829473.474290:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790714880:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829473.474298:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829473.474302:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829473.474303:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829473.474307:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6110 00000100:00080000:2.0:1787829473.474310:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829473.474312:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829473.474315:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829473.474323:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790715776/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829473.474333:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829473.474353:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829473/real 1787829473] req@ffff9e6804420b40 x1874668790714880/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829473.474384:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790714880/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829473.474401:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829473.474405:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829473.474410:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790714880:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829473.474440:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790715008:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829473.474465:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829473.474478:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829473/real 1787829473] req@ffff9e68044230c0 x1874668790715008/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829473.474491:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790715008/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829473.474500:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829473.474503:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829473.474507:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790715008:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829473.474514:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790715136:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829473.474537:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829473.474558:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829473/real 1787829473] req@ffff9e68044212c0 x1874668790715136/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829473.474573:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790715136/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829473.474581:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829473.474584:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829473.474588:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790715136:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829473.474596:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790715264:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829473.474618:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829473.474631:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829473/real 1787829473] req@ffff9e6804423840 x1874668790715264/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829473.474644:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790715264/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829473.474653:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829473.474656:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829473.474660:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790715264:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829473.474666:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790715392:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829473.474733:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829473.474745:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829473/real 1787829473] req@ffff9e68044221c0 x1874668790715392/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829473.474759:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790715392/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829473.474769:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829473.474773:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829473.474776:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790715392:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829473.474783:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790715520:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829473.474814:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829473.474826:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829473/real 1787829473] req@ffff9e6804420780 x1874668790715520/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829473.474839:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790715520/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829473.474847:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829473.474851:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829473.474855:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790715520:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829473.474861:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790715648:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829473.474891:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829473.474902:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829473/real 1787829473] req@ffff9e6804422d00 x1874668790715648/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829473.474915:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790715648/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829473.474925:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829473.474928:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829473.474931:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790715648:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829473.474937:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790715776:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829473.474953:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829473.474961:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829473/real 1787829473] req@ffff9e6804422940 x1874668790715776/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829473.474973:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790715776/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829473.474982:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829473.474985:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829473.474989:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790715776:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829473.985873:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829474.177876:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829475.009796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829475.201942:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829476.033876:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829476.225923:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829477.057837:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829477.249934:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829478.081844:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829478.273892:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829478.593767:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829478.593782:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829478.593787:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829478.593799:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6115 00000100:00080000:2.0:1787829478.593812:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829478.593821:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829478.593870:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790715904/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829478.593899:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829478.593904:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829478.593906:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829478.593910:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6115 00000100:00080000:2.0:1787829478.593914:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829478.593918:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829478.593930:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829478.593945:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790716032/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829478.593947:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790715904:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829478.593957:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829478.593960:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829478.593962:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829478.593964:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6115 00000100:00080000:2.0:1787829478.593967:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829478.593969:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829478.593973:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829478.593985:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790716160/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829478.593995:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829478.593998:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829478.594000:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829478.594004:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6115 00000100:00080000:2.0:1787829478.594008:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829478.594012:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829478.594018:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829478.594030:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790716288/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829478.594035:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829478.594046:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829478.594050:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829478.594052:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829478.594056:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6115 00000100:00080000:2.0:1787829478.594061:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829478.594065:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829478.594070:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829478.594082:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790716416/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829478.594091:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829478.594096:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829478/real 1787829478] req@ffff9e6804423480 x1874668790715904/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829478.594096:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829478.594100:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829478.594108:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6115 00000100:00080000:2.0:1787829478.594114:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829478.594119:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829478.594123:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829478.594129:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790715904/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829478.594132:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790716544/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829478.594156:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829478.594159:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829478.594160:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829478.594162:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829478.594170:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829478.594171:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6115 00000100:00080000:2.0:1787829478.594177:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829478.594179:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790715904:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829478.594184:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829478.594191:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829478.594202:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790716672/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829478.594212:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790716032:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829478.594213:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829478.594221:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829478.594223:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829478.594228:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6115 00000100:00080000:2.0:1787829478.594232:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829478.594236:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829478.594241:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829478.594247:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829478.594255:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790716800/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829478.594266:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829478.594270:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829478.594271:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829478/real 1787829478] req@ffff9e6804421680 x1874668790716032/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829478.594272:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829478.594279:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6115 00000100:00080000:2.0:1787829478.594286:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829478.594290:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829478.594295:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829478.594296:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790716032/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829478.594307:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790716928/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829478.594310:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829478.594319:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829478.594325:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790716032:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829478.594345:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790716160:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829478.594364:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829478.594375:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829478/real 1787829478] req@ffff9e6804422580 x1874668790716160/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829478.594386:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790716160/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829478.594393:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829478.594396:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829478.594400:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790716160:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829478.594404:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790716288:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829478.594418:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829478.594426:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829478/real 1787829478] req@ffff9e6804420f00 x1874668790716288/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829478.594436:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790716288/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829478.594443:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829478.594446:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829478.594449:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790716288:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829478.594453:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790716416:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829478.594466:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829478.594474:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829478/real 1787829478] req@ffff9e6804422940 x1874668790716416/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829478.594484:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790716416/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829478.594489:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829478.594491:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829478.594494:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790716416:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829478.594498:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790716544:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829478.594518:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829478.594529:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829478/real 1787829478] req@ffff9e6804422d00 x1874668790716544/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829478.594540:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790716544/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829478.594549:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829478.594552:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829478.594556:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790716544:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829478.594572:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790716672:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829478.594588:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829478.594597:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829478/real 1787829478] req@ffff9e6804420780 x1874668790716672/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829478.594613:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790716672/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829478.594620:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829478.594623:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829478.594626:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790716672:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829478.594630:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790716800:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829478.594644:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829478.594650:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829478/real 1787829478] req@ffff9e68044221c0 x1874668790716800/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829478.594657:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790716800/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829478.594662:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829478.594664:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829478.594665:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790716800:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829478.594668:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790716928:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829478.594677:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829478.594682:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829478/real 1787829478] req@ffff9e6804423840 x1874668790716928/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829478.594688:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790716928/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829478.594696:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829478.594715:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829478.594717:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790716928:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829479.105918:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829479.297855:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829480.129828:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829480.321866:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829481.153794:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829481.345850:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829482.177815:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829482.369874:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829483.201801:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829483.393904:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829483.713829:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829483.713840:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829483.713845:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829483.713857:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6120 00000100:00080000:2.0:1787829483.713870:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829483.713877:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829483.713918:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790717056/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829483.713945:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829483.713948:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829483.713950:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829483.713954:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6120 00000100:00080000:2.0:1787829483.713960:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829483.713965:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829483.713977:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790717056:192.168.203.109@tcp:250:kworker.0 00000008:00100000:2.0:1787829483.713977:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829483.713998:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790717184/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829483.714011:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829483.714014:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829483.714015:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829483.714019:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6120 00000100:00080000:2.0:1787829483.714023:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829483.714027:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829483.714033:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829483.714046:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790717312/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829483.714056:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829483.714059:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829483.714061:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829483.714067:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6120 00000100:00080000:2.0:1787829483.714072:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829483.714078:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829483.714082:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829483.714085:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829483.714095:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790717440/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829483.714105:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829483.714108:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829483.714111:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829483.714116:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6120 00000100:00080000:2.0:1787829483.714118:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829483.714121:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829483.714125:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829483.714137:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790717568/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829483.714144:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829483.714149:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787829483.714150:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829483/real 1787829483] req@ffff9e68044212c0 x1874668790717056/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829483.714151:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829483.714157:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6120 00000100:00080000:2.0:1787829483.714164:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829483.714175:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829483.714188:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829483.714201:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790717696/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829483.714208:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790717056/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829483.714212:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829483.714227:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829483.714233:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829483.714247:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6120 00000100:00080000:2.0:1787829483.714250:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829483.714251:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829483.714256:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787829483.714262:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:2.0:1787829483.714266:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829483.714270:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790717056:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829483.714282:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790717824/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829483.714288:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829483.714290:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829483.714307:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829483.714312:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6120 00000100:00080000:2.0:1787829483.714315:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829483.714318:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829483.714325:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829483.714329:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790717184:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829483.714338:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790717952/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829483.714347:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829483.714350:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829483.714351:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829483.714354:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6120 00000100:00080000:2.0:1787829483.714358:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829483.714362:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1787829483.714370:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:2.0:1787829483.714372:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829483.714387:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790718080/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829483.714396:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829483/real 1787829483] req@ffff9e68044230c0 x1874668790717184/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829483.714415:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790717184/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829483.714434:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829483.714439:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829483.714444:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790717184:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829483.714454:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790717312:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829483.714484:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829483.714515:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829483/real 1787829483] req@ffff9e6804420b40 x1874668790717312/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829483.714539:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790717312/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829483.714548:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829483.714552:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829483.714556:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790717312:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829483.714563:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790717440:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829483.714585:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829483.714593:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829483/real 1787829483] req@ffff9e6804421e00 x1874668790717440/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829483.714600:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790717440/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829483.714605:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829483.714607:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829483.714610:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790717440:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829483.714615:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790717568:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829483.714626:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829483.714631:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829483/real 1787829483] req@ffff9e6804423840 x1874668790717568/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829483.714638:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790717568/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829483.714643:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829483.714645:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829483.714646:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790717568:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829483.714650:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790717696:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829483.714660:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829483.714665:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829483/real 1787829483] req@ffff9e68044221c0 x1874668790717696/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829483.714671:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790717696/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829483.714678:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829483.714679:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829483.714681:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790717696:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829483.714684:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790717824:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829483.714693:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829483.714738:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829483/real 1787829483] req@ffff9e6804420780 x1874668790717824/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829483.714750:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790717824/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829483.714758:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829483.714762:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829483.714767:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790717824:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829483.714806:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790717952:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829483.714826:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829483.714836:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829483/real 1787829483] req@ffff9e6804422d00 x1874668790717952/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829483.714847:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790717952/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829483.714854:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829483.714857:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829483.714860:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790717952:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829483.714866:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790718080:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829483.714885:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829483.714895:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829483/real 1787829483] req@ffff9e6804422940 x1874668790718080/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829483.714906:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790718080/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829483.714914:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829483.714918:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829483.714921:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790718080:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829484.225842:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829484.417859:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829485.249891:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829485.441914:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829486.273808:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829486.465869:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829487.297825:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829487.489877:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829488.321821:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829488.513845:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829488.833793:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829488.833807:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829488.833812:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829488.833826:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6125 00000100:00080000:2.0:1787829488.833839:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829488.833847:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829488.833895:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790718208/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829488.833925:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829488.833929:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829488.833931:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829488.833936:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6125 00000100:00080000:2.0:1787829488.833939:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829488.833944:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829488.833956:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829488.833971:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790718336/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829488.833973:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790718208:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829488.833982:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829488.833991:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829488.833993:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829488.833997:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6125 00000100:00080000:2.0:1787829488.834001:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829488.834005:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829488.834010:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829488.834021:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790718464/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829488.834031:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829488.834034:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829488.834037:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829488.834040:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6125 00000100:00080000:2.0:1787829488.834045:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829488.834048:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829488.834054:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829488.834058:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829488.834067:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790718592/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829488.834077:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829488.834080:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829488.834082:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829488.834086:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6125 00000100:00080000:2.0:1787829488.834090:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829488.834094:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829488.834101:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829488/real 1787829488] req@ffff9e6804420f00 x1874668790718208/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829488.834106:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829488.834118:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790718208/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829488.834123:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790718720/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829488.834129:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829488.834136:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829488.834137:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829488.834140:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790718208:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829488.834144:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829488.834147:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829488.834152:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6125 00000100:00080000:2.0:1787829488.834157:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829488.834161:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829488.834165:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790718336:192.168.203.109@tcp:38:kworker.0 00000008:00100000:2.0:1787829488.834167:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829488.834184:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790718848/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829488.834186:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829488.834198:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829488.834200:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829488/real 1787829488] req@ffff9e6804422580 x1874668790718336/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829488.834204:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829488.834207:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829488.834216:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6125 00000100:00080000:2.0:1787829488.834221:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829488.834224:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790718336/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829488.834228:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829488.834237:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829488.834240:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829488.834246:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829488.834251:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790718976/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829488.834253:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790718336:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829488.834271:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790718464:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829488.834284:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829488.834288:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829488.834290:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829488.834293:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829488.834298:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6125 00000100:00080000:2.0:1787829488.834304:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829488.834309:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829488/real 1787829488] req@ffff9e6804421680 x1874668790718464/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829488.834310:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829488.834320:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829488.834323:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790718464/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829488.834330:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829488.834332:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829488.834334:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790718464:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829488.834341:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790718592:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829488.834347:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790719104/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829488.834363:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829488.834364:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829488.834369:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829488.834371:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829488.834376:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829488/real 1787829488] req@ffff9e6804423480 x1874668790718592/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829488.834377:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6125 00000100:00080000:2.0:1787829488.834384:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829488.834391:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829488.834395:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790718592/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829488.834397:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829488.834407:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829488.834410:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829488.834415:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790718592:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829488.834416:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790719232/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829488.834422:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790718720:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829488.834444:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829488.834452:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829488/real 1787829488] req@ffff9e6804422940 x1874668790718720/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829488.834459:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790718720/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829488.834464:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829488.834467:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829488.834468:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790718720:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829488.834481:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790718848:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829488.834498:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829488.834508:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829488/real 1787829488] req@ffff9e6804422d00 x1874668790718848/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829488.834520:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790718848/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829488.834529:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829488.834532:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829488.834534:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790718848:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829488.834538:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790718976:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829488.834551:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829488.834557:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829488/real 1787829488] req@ffff9e6804420780 x1874668790718976/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829488.834568:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790718976/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829488.834573:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829488.834574:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829488.834576:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790718976:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829488.834579:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790719104:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829488.834588:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829488.834593:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829488/real 1787829488] req@ffff9e68044221c0 x1874668790719104/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829488.834600:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790719104/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829488.834604:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829488.834606:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829488.834608:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790719104:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829488.834610:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790719232:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829488.834618:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829488.834623:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829488/real 1787829488] req@ffff9e6804423840 x1874668790719232/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829488.834628:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790719232/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829488.834637:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829488.834639:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829488.834641:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790719232:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829489.345844:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829489.537879:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829490.369803:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829490.561859:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829491.393832:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829491.585866:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829492.417785:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829492.609891:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829493.441839:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829493.633924:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829493.953782:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829493.953796:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829493.953801:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829493.953815:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6130 00000100:00080000:2.0:1787829493.953829:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829493.953838:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829493.953889:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790719360/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829493.953919:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829493.953924:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829493.953926:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829493.953931:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6130 00000100:00080000:2.0:1787829493.953935:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829493.953940:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829493.953952:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829493.953965:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790719360:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829493.953971:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790719488/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829493.953984:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829493.953987:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829493.953989:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829493.953993:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6130 00000100:00080000:2.0:1787829493.953997:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829493.954001:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829493.954006:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829493.954018:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790719616/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829493.954028:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829493.954031:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829493.954034:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829493.954038:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6130 00000100:00080000:2.0:1787829493.954042:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829493.954046:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829493.954051:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829493.954052:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829493.954065:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790719744/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829493.954076:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829493.954079:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829493.954081:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829493.954084:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6130 00000100:00080000:2.0:1787829493.954087:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829493.954091:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829493.954095:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829493.954108:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829493/real 1787829493] req@ffff9e6804421e00 x1874668790719360/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829493.954110:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790719872/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829493.954141:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829493.954145:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829493.954147:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829493.954152:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790719360/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829493.954156:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6130 00000100:00080000:2.0:1787829493.954165:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829493.954171:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829493.954171:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787829493.954180:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:2.0:1787829493.954181:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829493.954187:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790719360:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829493.954198:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790720000/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829493.954209:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829493.954212:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829493.954214:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829493.954220:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790719488:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829493.954233:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6130 00000100:00080000:2.0:1787829493.954241:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829493.954245:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829493.954251:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829493.954261:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829493.954266:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790720128/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829493.954278:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829493.954279:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829493/real 1787829493] req@ffff9e6804420b40 x1874668790719488/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829493.954287:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829493.954292:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829493.954301:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6130 00000100:00080000:2.0:1787829493.954306:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829493.954312:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790719488/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829493.954313:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829493.954323:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829493.954337:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787829493.954342:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790720256/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829493.954345:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829493.954352:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790719488:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829493.954364:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829493.954365:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790719616:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829493.954372:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829493.954376:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829493.954382:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6130 00000100:00080000:2.0:1787829493.954387:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829493.954395:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1787829493.954396:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:2.0:1787829493.954405:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829493.954414:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829493/real 1787829493] req@ffff9e68044230c0 x1874668790719616/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829493.954420:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790720384/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829493.954435:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790719616/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829493.954445:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829493.954450:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829493.954455:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790719616:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829493.954462:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790719744:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829493.954481:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829493.954491:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829493/real 1787829493] req@ffff9e68044212c0 x1874668790719744/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829493.954503:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790719744/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829493.954510:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829493.954513:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829493.954517:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790719744:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829493.954522:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790719872:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829493.954541:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829493.954552:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829493/real 1787829493] req@ffff9e6804423840 x1874668790719872/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829493.954564:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790719872/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829493.954572:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829493.954575:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829493.954578:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790719872:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829493.954584:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790720000:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829493.954605:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829493.954616:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829493/real 1787829493] req@ffff9e68044221c0 x1874668790720000/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829493.954628:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790720000/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829493.954638:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829493.954641:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829493.954644:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790720000:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829493.954662:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790720128:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829493.954689:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829493.954723:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829493/real 1787829493] req@ffff9e6804420780 x1874668790720128/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829493.954735:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790720128/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829493.954742:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829493.954745:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829493.954748:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790720128:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829493.954754:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790720256:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829493.954772:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829493.954783:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829493/real 1787829493] req@ffff9e6804422d00 x1874668790720256/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829493.954794:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790720256/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829493.954800:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829493.954803:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829493.954806:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790720256:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829493.954812:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790720384:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829493.954835:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829493.954846:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829493/real 1787829493] req@ffff9e6804422940 x1874668790720384/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829493.954856:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790720384/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829518 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829493.954864:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829493.954867:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829493.954870:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790720384:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829494.465798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829494.657876:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829495.489769:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829495.681881:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829496.513772:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829496.705851:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829497.537801:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829497.729905:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829498.561809:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829498.753839:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829499.073796:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829499.073807:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829499.073812:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829499.073821:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6135 00000100:00080000:2.0:1787829499.073830:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829499.073844:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829499.073881:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790720512/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829499.073913:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829499.073916:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829499.073918:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829499.073921:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6135 00000100:00080000:2.0:1787829499.073924:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829499.073928:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829499.073936:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829499.073948:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790720640/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829499.073956:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829499.073959:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829499.073963:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829499.073964:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790720512:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829499.073968:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6135 00000100:00080000:2.0:1787829499.073974:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829499.073977:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829499.073981:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829499.073997:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790720768/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829499.074005:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829499.074008:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829499.074009:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829499.074011:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6135 00000100:00080000:2.0:1787829499.074014:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829499.074017:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829499.074020:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829499.074028:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790720896/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829499.074035:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829499.074037:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829499.074039:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829499.074041:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6135 00000100:00080000:2.0:1787829499.074043:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829499.074046:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829499.074051:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829499.074063:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790720640:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829499.074064:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790721024/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829499.074078:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829499.074081:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829499.074083:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829499.074088:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6135 00000100:00080000:2.0:1787829499.074091:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829499.074095:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829499.074099:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829499.074100:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790720768:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829499.074114:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790721152/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829499.074122:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829499.074126:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790720896:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787829499.074127:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829499.074130:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829499.074137:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6135 00000100:00080000:2.0:1787829499.074140:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829499.074143:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829499.074147:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829499.074156:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790721024:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829499.074160:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790721280/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829499.074169:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829499.074171:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829499.074173:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829499.074178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790721152:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829499.074179:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6135 00000100:00080000:2.0:1787829499.074200:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829499.074205:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829499.074211:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829499.074222:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790721408/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829499.074225:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790721280:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829499.074240:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829499.074244:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829499.074245:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829499.074250:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6135 00000100:00080000:2.0:1787829499.074253:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829499.074256:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790721408:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787829499.074260:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829499.074266:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829499.074277:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790721536/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829499.074308:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790721536:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829499.585796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829499.777846:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829500.609790:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829500.801861:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829501.633802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829501.825850:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787829502.145963:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:1.0:1787829502.146023:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787829502.146043:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829502.146104:0:2386:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:1.0:1787829502.146119:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829502.146132:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829502.146144:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829502.146154:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829502.146182:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829502.146198:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829502.146215:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787829502.146215:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829499/real 1787829502] req@ffff9e6804423840 x1874668790721536/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829502.146262:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790721536/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829502.146274:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829502.146280:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829502.146285:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790721536:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829502.146293:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829499/real 1787829502] req@ffff9e68044221c0 x1874668790721408/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829502.146305:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790721408/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829502.146312:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829502.146315:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829502.146319:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790721408:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829502.146340:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829499/real 1787829502] req@ffff9e6804420780 x1874668790721280/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829502.146353:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790721280/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829502.146361:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829502.146365:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829502.146369:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790721280:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829502.146381:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829499/real 1787829502] req@ffff9e6804422940 x1874668790721024/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829502.146393:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790721024/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829502.146402:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829502.146405:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829502.146408:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790721024:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829502.146415:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829499/real 1787829502] req@ffff9e6804422d00 x1874668790721152/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829502.146437:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790721152/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829502.146448:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829502.146452:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829502.146461:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790721152:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829502.146468:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829499/real 1787829502] req@ffff9e6804421680 x1874668790720640/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829502.146481:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790720640/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829502.146490:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829502.146494:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829502.146497:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790720640:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829502.146504:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829499/real 1787829502] req@ffff9e6804422580 x1874668790720768/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829502.146515:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790720768/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829502.146523:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829502.146527:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829502.146531:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790720768:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829502.146538:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829499/real 1787829502] req@ffff9e6804420f00 x1874668790720896/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829502.146549:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790720896/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829502.146558:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829502.146561:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829502.146565:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790720896:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829502.146573:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829499/real 1787829502] req@ffff9e6804423480 x1874668790720512/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829502.146585:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790720512/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829502.146591:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829502.146593:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829502.146595:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790720512:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829502.657817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829502.849894:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829503.681819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829503.873869:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829504.193816:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829504.193824:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829504.193828:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829504.193836:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6140 00000100:00080000:1.0:1787829504.193846:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829504.193854:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829504.193888:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790721664/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829504.193917:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829504.193921:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829504.193923:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829504.193928:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6140 00000100:00080000:1.0:1787829504.193933:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829504.193938:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829504.193948:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829504.193960:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790721664:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787829504.193964:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790721792/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829504.193976:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829504.193979:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829504.193980:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829504.193983:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6140 00000100:00080000:1.0:1787829504.193986:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829504.193989:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829504.193992:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829504.193999:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790721920/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829504.194007:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829504.194011:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829504.194013:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829504.194017:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6140 00000100:00080000:1.0:1787829504.194021:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829504.194025:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829504.194031:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829504.194045:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790722048/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829504.194046:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829504.194058:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829504.194062:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829504.194064:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829504.194068:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6140 00000100:00080000:1.0:1787829504.194072:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829504.194077:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829504.194083:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829504.194097:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829504/real 1787829504] req@ffff9e6831aab480 x1874668790721664/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829504.194099:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790722176/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829504.194134:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829504.194138:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829504.194141:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829504.194146:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790721664/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829504.194148:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6140 00000100:00080000:1.0:1787829504.194166:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829504.194171:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829504.194175:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829504.194186:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790722304/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829504.194194:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829504.194197:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829504.194198:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829504.194201:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6140 00000100:00080000:1.0:1787829504.194204:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829504.194207:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829504.194211:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829504.194218:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790722432/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829504.194226:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829504.194228:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829504.194230:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829504.194233:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6140 00000100:00080000:1.0:1787829504.194236:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829504.194239:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829504.194243:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829504.194249:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790722560/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829504.194257:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829504.194260:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829504.194261:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829504.194265:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6140 00000100:00080000:1.0:1787829504.194267:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829504.194271:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829504.194278:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829504.194289:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790722688/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829504.194293:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829504.194305:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829504.194311:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790721664:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829504.194339:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790721792:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829504.194368:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829504.194383:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829504/real 1787829504] req@ffff9e6831aa8780 x1874668790721792/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829504.194396:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790721792/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829504.194405:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829504.194409:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829504.194412:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790721792:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829504.194418:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790721920:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829504.194439:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829504.194451:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829504/real 1787829504] req@ffff9e6831aaa1c0 x1874668790721920/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829504.194462:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790721920/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829504.194469:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829504.194473:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829504.194476:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790721920:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829504.194482:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790722048:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829504.194504:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829504.194515:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829504/real 1787829504] req@ffff9e6831aaad00 x1874668790722048/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829504.194545:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790722048/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829504.194554:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829504.194558:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829504.194561:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790722048:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829504.194567:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790722176:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829504.194589:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829504.194601:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829504/real 1787829504] req@ffff9e6831aa8000 x1874668790722176/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829504.194613:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790722176/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829504.194621:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829504.194624:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829504.194627:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790722176:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829504.194633:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790722304:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829504.194655:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829504.194667:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829504/real 1787829504] req@ffff9e6831aaa940 x1874668790722304/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829504.194681:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790722304/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829504.194692:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829504.194721:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829504.194725:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790722304:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829504.194732:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790722432:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829504.194758:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829504.194773:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829504/real 1787829504] req@ffff9e6831aabc00 x1874668790722432/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829504.194787:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790722432/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829504.194795:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829504.194798:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829504.194801:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790722432:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829504.194807:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790722560:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829504.194831:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829504.194857:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829504/real 1787829504] req@ffff9e6831aa9a40 x1874668790722560/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829504.194869:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790722560/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829504.194888:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829504.194891:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829504.194895:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790722560:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829504.194901:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790722688:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829504.194921:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829504.194933:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829504/real 1787829504] req@ffff9e6831aa9680 x1874668790722688/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829504.194945:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790722688/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829504.194953:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829504.194956:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829504.194960:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790722688:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829504.705777:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829504.897862:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829505.729770:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829505.921826:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829506.753782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829506.945835:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829507.777799:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829507.969844:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829508.801824:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829508.993864:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829509.313768:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829509.313778:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829509.313782:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829509.313792:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6146 00000100:00080000:1.0:1787829509.313803:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829509.313809:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829509.313856:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790722816/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829509.313878:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829509.313881:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829509.313882:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829509.313885:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6146 00000100:00080000:1.0:1787829509.313887:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829509.313889:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829509.313898:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829509.313910:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790722944/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829509.313916:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790722816:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829509.313919:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829509.313924:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829509.313925:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829509.313928:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6146 00000100:00080000:1.0:1787829509.313930:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829509.313932:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829509.313934:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829509.313943:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790723072/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829509.313949:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829509.313951:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829509.313952:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829509.313955:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6146 00000100:00080000:1.0:1787829509.313958:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829509.313960:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829509.313963:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829509.313971:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790723200/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829509.313976:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829509.313978:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829509.313980:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829509.313982:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6146 00000100:00080000:1.0:1787829509.313985:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829509.313988:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829509.313990:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829509.313997:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829509.313999:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790723328/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829509.314006:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829509.314009:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829509.314011:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829509.314013:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6146 00000100:00080000:1.0:1787829509.314019:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829509.314021:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829509.314023:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829509.314030:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790723456/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829509.314040:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829509.314043:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829509/real 1787829509] req@ffff9e6831aa8b40 x1874668790722816/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829509.314044:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829509.314047:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829509.314054:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6146 00000100:00080000:1.0:1787829509.314058:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829509.314061:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829509.314064:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829509.314069:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790722816/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829509.314072:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790723584/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829509.314091:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829509.314092:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829509.314096:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829509.314099:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829509.314100:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829509.314105:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790722816:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829509.314106:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6146 00000100:00080000:1.0:1787829509.314112:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829509.314129:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829509.314132:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829509.314139:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790723712/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829509.314146:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790722944:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829509.314148:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829509.314152:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829509.314153:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829509.314157:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6146 00000100:00080000:1.0:1787829509.314160:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829509.314163:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829509.314167:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829509.314171:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829509.314177:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790723840/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829509.314184:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829509/real 1787829509] req@ffff9e6831aa9e00 x1874668790722944/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829509.314194:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790722944/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829509.314206:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829509.314208:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829509.314210:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790722944:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829509.314214:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790723072:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829509.314227:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829509.314233:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829509/real 1787829509] req@ffff9e6831aa8f00 x1874668790723072/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829509.314239:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790723072/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829509.314244:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829509.314246:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829509.314247:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790723072:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829509.314251:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790723200:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829509.314262:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829509.314267:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829509/real 1787829509] req@ffff9e6831aaa580 x1874668790723200/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829509.314274:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790723200/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829509.314279:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829509.314280:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829509.314282:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790723200:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829509.314285:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790723328:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829509.314293:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829509.314305:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829509/real 1787829509] req@ffff9e6831aa9680 x1874668790723328/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829509.314312:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790723328/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829509.314317:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829509.314319:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829509.314320:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790723328:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829509.314323:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790723456:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829509.314335:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829509.314344:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829509/real 1787829509] req@ffff9e6831aa9a40 x1874668790723456/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829509.314356:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790723456/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829509.314366:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829509.314369:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829509.314372:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790723456:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829509.314378:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790723584:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829509.314392:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829509.314399:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829509/real 1787829509] req@ffff9e6831aabc00 x1874668790723584/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829509.314406:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790723584/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829509.314411:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829509.314413:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829509.314415:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790723584:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829509.314427:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790723712:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829509.314436:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829509.314441:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829509/real 1787829509] req@ffff9e6831aaa940 x1874668790723712/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829509.314448:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790723712/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829509.314456:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829509.314458:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829509.314460:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790723712:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829509.314463:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790723840:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829509.314471:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829509.314476:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829509/real 1787829509] req@ffff9e6831aa8000 x1874668790723840/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829509.314482:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790723840/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829509.314487:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829509.314488:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829509.314490:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790723840:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829509.825798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829510.017897:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829510.849818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829511.041828:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829511.873819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829512.065898:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829512.897816:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829513.089882:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829513.921761:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829514.113907:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829514.433839:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829514.433854:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829514.433860:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829514.433871:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6151 00000100:00080000:1.0:1787829514.433885:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829514.433895:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829514.433943:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790723968/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829514.433977:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829514.433982:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829514.433984:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829514.433989:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6151 00000100:00080000:1.0:1787829514.433992:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829514.433997:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829514.434008:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829514.434024:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790724096/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829514.434025:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790723968:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829514.434039:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829514.434041:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829514.434042:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829514.434044:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6151 00000100:00080000:1.0:1787829514.434046:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829514.434048:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829514.434051:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829514.434059:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790724224/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829514.434065:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829514.434067:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829514.434068:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829514.434070:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6151 00000100:00080000:1.0:1787829514.434073:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829514.434075:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829514.434090:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829514.434100:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790724352/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829514.434106:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829514.434108:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829514.434109:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829514.434114:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6151 00000800:00000100:2.0:1787829514.434115:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829514.434119:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829514.434124:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829514.434130:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829514.434141:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790724480/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829514.434151:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829514.434155:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829514.434157:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829514.434164:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6151 00000100:00100000:2.0:1787829514.434165:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829514/real 1787829514] req@ffff9e6831aaad00 x1874668790723968/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829514.434168:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829514.434177:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829514.434184:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829514.434200:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790724608/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829514.434200:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790723968/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829514.434217:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829514.434221:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829514.434222:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829514.434227:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6151 00000100:00080000:1.0:1787829514.434233:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829514.434235:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829514.434240:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787829514.434245:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:1.0:1787829514.434247:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829514.434250:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790723968:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787829514.434262:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790724736/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829514.434269:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829514.434272:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829514.434274:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829514.434277:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6151 00000100:00100000:2.0:1787829514.434280:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790724096:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829514.434282:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829514.434290:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829514.434296:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829514.434306:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829514.434312:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790724864/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829514.434319:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829514/real 1787829514] req@ffff9e6831aaa1c0 x1874668790724096/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829514.434329:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829514.434334:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829514.434336:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829514.434341:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6151 00000100:00100000:2.0:1787829514.434342:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790724096/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829514.434344:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829514.434349:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829514.434356:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829514.434360:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829514.434367:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829514.434370:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790724992/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829514.434375:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790724096:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829514.434389:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790724224:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829514.434414:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829514.434424:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829514/real 1787829514] req@ffff9e6831aa8780 x1874668790724224/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829514.434435:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790724224/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829514.434443:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829514.434447:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829514.434450:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790724224:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829514.434456:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790724352:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829514.434487:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829514.434500:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829514/real 1787829514] req@ffff9e6831aab480 x1874668790724352/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829514.434510:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790724352/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829514.434516:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829514.434519:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829514.434521:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790724352:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829514.434524:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790724480:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829514.434545:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829514.434551:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829514/real 1787829514] req@ffff9e6831aa8000 x1874668790724480/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829514.434558:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790724480/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829514.434563:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829514.434566:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829514.434567:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790724480:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829514.434570:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790724608:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829514.434580:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829514.434585:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829514/real 1787829514] req@ffff9e6831aaa940 x1874668790724608/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829514.434592:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790724608/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829514.434604:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829514.434606:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829514.434608:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790724608:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829514.434611:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790724736:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829514.434622:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829514.434629:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829514/real 1787829514] req@ffff9e6831aabc00 x1874668790724736/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829514.434640:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790724736/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829514.434647:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829514.434650:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829514.434654:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790724736:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829514.434674:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790724864:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829514.434693:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829514.434737:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829514/real 1787829514] req@ffff9e6831aa9a40 x1874668790724864/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829514.434745:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790724864/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829514.434757:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829514.434761:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829514.434764:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790724864:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829514.434769:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790724992:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829514.434789:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829514.434809:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829514/real 1787829514] req@ffff9e6831aa9680 x1874668790724992/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829514.434816:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790724992/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829514.434822:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829514.434824:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829514.434826:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790724992:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829514.945823:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829515.137924:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829515.969823:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829516.161882:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829516.993819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829517.185855:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829518.017835:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829518.209855:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829519.041839:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829519.233989:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829519.553831:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829519.553839:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829519.553844:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829519.553854:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6156 00000100:00080000:3.0:1787829519.553864:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829519.553872:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829519.553915:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790725120/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829519.553937:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829519.553940:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829519.553942:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829519.553945:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6156 00000100:00080000:3.0:1787829519.553949:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829519.553952:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829519.553960:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829519.553968:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790725120:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829519.553977:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790725248/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829519.553988:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829519.553991:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829519.553993:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829519.553996:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6156 00000100:00080000:3.0:1787829519.553999:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829519.554003:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829519.554006:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829519.554021:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790725376/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829519.554029:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829519.554031:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829519.554033:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829519.554036:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6156 00000100:00080000:3.0:1787829519.554063:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829519.554069:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829519.554075:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829519.554081:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829519.554141:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829519/real 1787829519] req@ffff9e680a2d5a40 x1874668790725120/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829519.554154:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790725504/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829519.554168:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829519.554169:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790725120/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829519.554188:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829519.554192:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829519.554196:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790725120:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829519.554227:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790725248:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829519.554251:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829519.554264:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829519/real 1787829519] req@ffff9e680a2d7480 x1874668790725248/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829519.554276:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790725248/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829519.554284:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829519.554291:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829519.554293:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829519.554306:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829519.554307:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6156 00000100:00100000:2.0:1787829519.554319:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790725248:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829519.554325:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790725376:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829519.554345:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829519.554362:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829519.554366:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829519/real 1787829519] req@ffff9e680a2d4b40 x1874668790725376/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829519.554377:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829519.554386:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790725376/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829519.554388:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829519.554404:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829519.554409:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829519.554411:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790725376:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829519.554419:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790725504:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829519.554421:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790725632/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829519.554435:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829519.554439:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829519.554446:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829519/real 1787829519] req@ffff9e680a2d4f00 x1874668790725504/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829519.554447:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829519.554454:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829519.554460:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790725504/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829519.554465:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6156 00000100:00080000:2.0:1787829519.554471:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829519.554475:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829519.554477:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790725504:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829519.554487:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829519.554495:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829519.554497:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790725632:192.168.203.109@tcp:8:kworker.0 00000008:00100000:3.0:1787829519.554507:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829519.554518:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829519.554522:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790725760/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829519.554528:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829519/real 1787829519] req@ffff9e680a2d7c00 x1874668790725632/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829519.554560:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829519.554566:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787829519.554570:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790725632/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829519.554571:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829519.554585:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6156 00000100:00080000:3.0:1787829519.554596:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829519.554598:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829519.554604:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787829519.554607:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:3.0:1787829519.554611:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829519.554617:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790725632:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829519.554626:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790725888/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829519.554636:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829519.554638:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790725760:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829519.554641:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829519.554646:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829519.554651:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6156 00000100:00080000:3.0:1787829519.554654:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829519.554657:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829519.554662:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829519.554673:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790726016/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829519.554680:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829519.554692:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829519.554715:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829519.554717:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829519.554723:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6156 00000100:00080000:3.0:1787829519.554728:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829519.554733:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829519/real 1787829519] req@ffff9e680a2d4f00 x1874668790725760/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829519.554739:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829519.554751:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829519.554762:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790725760/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829519.554770:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790726144/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829519.554779:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829519.554787:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829519.554794:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790725760:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829519.554811:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790725888:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829519.554835:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829519.554847:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829519/real 1787829519] req@ffff9e680a2d4b40 x1874668790725888/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829519.554859:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790725888/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829519.554866:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829519.554869:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829519.554872:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790725888:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829519.554879:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790726016:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829519.554895:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829519.554904:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829519/real 1787829519] req@ffff9e680a2d7480 x1874668790726016/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829519.554915:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790726016/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829519.554929:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829519.554932:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829519.554935:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790726016:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829519.554940:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790726144:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829519.554953:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829519.554960:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829519/real 1787829519] req@ffff9e680a2d5a40 x1874668790726144/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829519.554970:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790726144/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829519.554977:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829519.554979:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829519.554982:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790726144:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829520.065823:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829520.257877:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829521.089826:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829521.281880:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829522.113779:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829522.306260:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829523.137804:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829523.329897:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829524.161815:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829524.353910:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829524.673804:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829524.673813:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829524.673816:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829524.673825:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6161 00000100:00080000:1.0:1787829524.673842:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829524.673848:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829524.673882:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790726272/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829524.673903:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829524.673906:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829524.673907:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829524.673910:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6161 00000100:00080000:1.0:1787829524.673913:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829524.673915:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829524.673924:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829524.673932:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790726400/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829524.673940:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829524.673942:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829524.673943:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829524.673946:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6161 00000100:00080000:1.0:1787829524.673948:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829524.673951:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829524.673955:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829524.673966:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790726528/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829524.673972:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790726272:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829524.673975:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829524.673985:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829524.673990:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829524.673996:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6161 00000100:00080000:1.0:1787829524.673998:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829524.674001:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829524.674004:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829524.674027:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790726656/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829524.674034:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829524.674036:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829524.674038:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829524.674040:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6161 00000100:00080000:1.0:1787829524.674043:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829524.674046:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829524.674049:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829524.674058:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790726784/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829524.674064:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829524.674066:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829524.674068:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829524.674070:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6161 00000100:00080000:1.0:1787829524.674079:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829524.674082:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829524.674085:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829524.674095:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790726912/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829524.674097:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829524.674114:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829524.674118:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829524.674120:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829524.674125:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6161 00000100:00080000:1.0:1787829524.674139:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829524.674144:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829524.674149:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829524.674163:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790727040/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829524.674172:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829524/real 1787829524] req@ffff9e6831aaa580 x1874668790726272/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829524.674175:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829524.674185:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829524.674187:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829524.674193:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6161 00000100:00080000:1.0:1787829524.674200:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829524.674205:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829524.674210:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829524.674227:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790727168/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829524.674227:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790726272/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829524.674268:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829524.674281:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829524.674283:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829524.674288:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6161 00000100:00080000:2.0:1787829524.674289:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829524.674293:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829524.674301:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787829524.674302:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:1.0:1787829524.674304:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829524.674316:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790726272:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787829524.674317:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790727296/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829524.674344:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790726400:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829524.674374:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829524.674387:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829524/real 1787829524] req@ffff9e6831aa8f00 x1874668790726400/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829524.674400:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790726400/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829524.674409:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829524.674412:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829524.674415:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790726400:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829524.674443:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790726528:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829524.674456:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829524.674463:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829524/real 1787829524] req@ffff9e6831aa9e00 x1874668790726528/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829524.674470:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790726528/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829524.674477:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829524.674480:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829524.674484:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790726528:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829524.674491:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790726656:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829524.674510:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829524.674520:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829524/real 1787829524] req@ffff9e6831aa8b40 x1874668790726656/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829524.674531:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790726656/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829524.674539:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829524.674543:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829524.674546:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790726656:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829524.674552:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790726784:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829524.674569:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829524.674579:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829524/real 1787829524] req@ffff9e6831aa9680 x1874668790726784/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829524.674589:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790726784/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829524.674597:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829524.674600:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829524.674604:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790726784:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829524.674610:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790726912:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829524.674627:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829524.674638:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829524/real 1787829524] req@ffff9e6831aa9a40 x1874668790726912/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829524.674649:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790726912/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829524.674659:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829524.674662:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829524.674665:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790726912:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829524.674670:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790727040:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829524.674688:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829524.674725:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829524/real 1787829524] req@ffff9e6831aabc00 x1874668790727040/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829524.674738:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790727040/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829524.674745:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829524.674748:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829524.674751:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790727040:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829524.674757:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790727168:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829524.674777:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829524.674787:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829524/real 1787829524] req@ffff9e6831aaa940 x1874668790727168/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829524.674798:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790727168/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829524.674812:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829524.674815:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829524.674818:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790727168:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829524.674824:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790727296:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829524.674842:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829524.674852:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829524/real 1787829524] req@ffff9e6831aa8000 x1874668790727296/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829524.674862:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790727296/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829524.674870:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829524.674873:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829524.674876:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790727296:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829525.185814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829525.379076:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829526.209839:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829526.401840:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829527.233814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829527.425876:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829528.257856:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829528.449926:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829529.281776:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829529.473878:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829529.793800:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829529.793809:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829529.793813:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829529.793824:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6166 00000100:00080000:0.0:1787829529.793834:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829529.793841:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829529.793879:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790727424/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829529.793901:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829529.793904:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829529.793906:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829529.793908:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6166 00000100:00080000:0.0:1787829529.793911:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829529.793913:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829529.793922:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829529.793931:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790727552/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829529.793938:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829529.793942:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829529.793944:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829529.793946:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6166 00000100:00080000:0.0:1787829529.793950:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829529.793951:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790727424:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829529.793954:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829529.793960:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829529.793966:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790727680/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829529.793982:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829529.793985:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829529.793986:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829529.793989:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6166 00000100:00080000:0.0:1787829529.793992:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829529.793995:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829529.793999:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829529.794007:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790727808/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829529.794013:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829529.794015:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829529.794017:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829529.794019:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6166 00000100:00080000:0.0:1787829529.794021:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829529.794023:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829529.794027:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829529.794031:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829529.794038:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790727936/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829529.794045:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829529.794048:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829529.794049:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829529.794052:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6166 00000100:00080000:0.0:1787829529.794055:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829529.794059:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829529.794063:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829529.794074:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829529/real 1787829529] req@ffff9e6803e9b480 x1874668790727424/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829529.794076:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790728064/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829529.794095:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829529.794099:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790727424/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829529.794101:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829529.794105:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829529.794113:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6166 00000100:00080000:2.0:1787829529.794114:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829529.794118:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829529.794119:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829529.794125:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829529.794126:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790727424:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1787829529.794131:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829529.794134:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790727552:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829529.794143:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790728192/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829529.794150:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829529.794153:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829529.794155:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829529.794157:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829529.794161:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6166 00000100:00080000:0.0:1787829529.794164:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829529.794166:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829529/real 1787829529] req@ffff9e6803e99e00 x1874668790727552/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829529.794169:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829529.794174:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829529.794181:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790727552/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829529.794186:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790728320/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829529.794193:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829529.794197:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829529.794198:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829529.794199:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829529.794205:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829529.794207:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6166 00000100:00080000:0.0:1787829529.794210:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829529.794212:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790727552:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829529.794214:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829529.794220:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829529.794227:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790728448/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829529.794253:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790727680:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829529.794275:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829529.794287:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829529/real 1787829529] req@ffff9e6803e99680 x1874668790727680/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829529.794298:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790727680/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829529.794306:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829529.794309:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829529.794313:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790727680:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829529.794319:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790727808:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829529.794338:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829529.794349:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829529/real 1787829529] req@ffff9e6803e9a580 x1874668790727808/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829529.794360:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790727808/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829529.794369:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829529.794372:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829529.794376:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790727808:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829529.794381:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790727936:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829529.794400:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829529.794410:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829529/real 1787829529] req@ffff9e6803e9b840 x1874668790727936/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829529.794422:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790727936/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829529.794430:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829529.794433:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829529.794436:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790727936:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829529.794442:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790728064:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829529.794460:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829529.794470:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829529/real 1787829529] req@ffff9e6803e9a1c0 x1874668790728064/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829529.794483:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790728064/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829529.794492:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829529.794495:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829529.794498:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790728064:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829529.794504:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790728192:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829529.794523:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829529.794535:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829529/real 1787829529] req@ffff9e6803e98780 x1874668790728192/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829529.794546:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790728192/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829529.794553:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829529.794557:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829529.794560:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790728192:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829529.794565:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790728320:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829529.794584:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829529.794594:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829529/real 1787829529] req@ffff9e6803e98b40 x1874668790728320/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829529.794605:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790728320/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829529.794619:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829529.794622:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829529.794626:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790728320:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829529.794632:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790728448:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829529.794650:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829529.794660:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829529/real 1787829529] req@ffff9e6803e9a940 x1874668790728448/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829529.794672:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790728448/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829529.794679:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829529.794682:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829529.794685:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790728448:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829530.305793:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829530.497850:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829531.329827:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829531.521936:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829532.353812:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829532.545852:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829533.377783:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829533.569880:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829534.401837:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829534.593898:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829534.913798:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829534.913810:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829534.913814:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829534.913824:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6171 00000100:00080000:3.0:1787829534.913835:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829534.913843:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829534.913898:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790728576/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829534.913928:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829534.913932:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829534.913934:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829534.913948:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6171 00000100:00080000:3.0:1787829534.913952:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829534.913957:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829534.913968:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829534.913981:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790728704/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829534.913992:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829534.913996:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829534.913997:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829534.914003:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790728576:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829534.914004:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6171 00000100:00080000:3.0:1787829534.914009:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829534.914015:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829534.914020:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829534.914033:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790728832/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829534.914043:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829534.914047:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829534.914049:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829534.914053:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6171 00000100:00080000:3.0:1787829534.914071:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829534.914075:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829534.914079:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829534.914091:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829534.914093:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790728960/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829534.914101:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829534.914104:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829534.914105:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829534.914108:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6171 00000100:00080000:3.0:1787829534.914111:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829534.914113:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829534.914115:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829534.914124:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790729088/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829534.914133:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829534.914137:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829534.914139:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829534.914145:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829534/real 1787829534] req@ffff9e680a2d5e00 x1874668790728576/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829534.914145:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6171 00000100:00080000:3.0:1787829534.914150:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829534.914158:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829534.914165:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829534.914171:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790728576/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829534.914176:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790729216/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829534.914193:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829534.914194:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829534.914197:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829534.914200:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829534.914202:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829534.914205:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6171 00000100:00080000:3.0:1787829534.914209:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829534.914211:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790728576:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829534.914214:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829534.914219:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829534.914226:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790729344/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829534.914231:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829534.914233:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829534.914234:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829534.914237:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6171 00000100:00080000:3.0:1787829534.914239:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829534.914243:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790728704:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829534.914250:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829534.914254:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829534.914263:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790729472/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829534.914274:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829534.914277:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829534.914281:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829534.914284:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829534.914289:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6171 00000100:00100000:2.0:1787829534.914292:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829534/real 1787829534] req@ffff9e680a2d6d00 x1874668790728704/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829534.914294:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829534.914303:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829534.914310:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829534.914312:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790728704/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829534.914323:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790729600/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829534.914324:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829534.914333:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829534.914338:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790728704:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829534.914345:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790728832:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829534.914365:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829534.914374:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829534/real 1787829534] req@ffff9e680a2d61c0 x1874668790728832/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829534.914385:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790728832/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829534.914393:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829534.914396:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829534.914398:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790728832:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829534.914404:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790728960:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829534.914420:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829534.914428:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829534/real 1787829534] req@ffff9e680a2d52c0 x1874668790728960/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829534.914437:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790728960/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829534.914444:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829534.914447:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829534.914450:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790728960:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829534.914454:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790729088:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829534.914469:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829534.914476:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829534/real 1787829534] req@ffff9e680a2d4780 x1874668790729088/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829534.914485:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790729088/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829534.914492:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829534.914495:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829534.914497:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790729088:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829534.914502:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790729216:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829534.914518:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829534.914527:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829534/real 1787829534] req@ffff9e680a2d43c0 x1874668790729216/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829534.914538:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790729216/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829534.914547:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829534.914550:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829534.914553:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790729216:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829534.914558:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790729344:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829534.914577:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829534.914587:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829534/real 1787829534] req@ffff9e680a2d70c0 x1874668790729344/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829534.914599:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790729344/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829534.914607:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829534.914610:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829534.914613:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790729344:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829534.914634:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790729472:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829534.914653:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829534.914664:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829534/real 1787829534] req@ffff9e680a2d6580 x1874668790729472/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829534.914675:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790729472/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829534.914690:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829534.914693:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829534.914726:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790729472:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829534.914730:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790729600:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829534.914746:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829534.914753:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829534/real 1787829534] req@ffff9e680a2d4000 x1874668790729600/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829534.914760:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790729600/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829559 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829534.914765:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829534.914767:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829534.914769:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790729600:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829535.425921:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829535.617905:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829536.449842:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829536.641817:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829537.473786:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829537.665848:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829538.497773:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829538.689893:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829539.521796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829539.713884:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829540.033825:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829540.033837:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829540.033842:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829540.033854:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6176 00000100:00080000:3.0:1787829540.033866:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829540.033874:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829540.033927:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790729728/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829540.033955:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829540.033960:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829540.033964:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829540.033969:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6176 00000100:00080000:3.0:1787829540.033974:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829540.033979:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829540.033989:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829540.033995:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790729728:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829540.034024:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790729856/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829540.034044:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829540.034050:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829540.034053:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829540.034059:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6176 00000100:00080000:3.0:1787829540.034066:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829540.034073:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787829540.034075:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787829540.034085:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829540.034102:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790729984/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829540.034116:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829540.034119:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829540.034121:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829540.034126:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6176 00000100:00080000:3.0:1787829540.034131:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829540.034139:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829540/real 1787829540] req@ffff9e680a2d4000 x1874668790729728/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829540.034140:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829540.034156:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829540.034164:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790729728/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829540.034175:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790730112/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829540.034179:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829540.034188:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829540.034200:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790729728:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829540.034217:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829540.034220:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829540.034222:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829540.034227:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6176 00000100:00080000:3.0:1787829540.034232:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829540.034238:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790729856:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829540.034240:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829540.034251:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829540.034264:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829540.034272:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790730240/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829540.034277:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829540/real 1787829540] req@ffff9e680a2d6580 x1874668790729856/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829540.034322:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790729856/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829540.034335:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829540.034336:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829540.034349:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829540.034351:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787829540.034354:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790729856:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829540.034357:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829540.034362:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790729984:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829540.034370:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6176 00000100:00080000:3.0:1787829540.034377:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829540.034385:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787829540.034387:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829540.034400:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829540/real 1787829540] req@ffff9e680a2d70c0 x1874668790729984/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829540.034400:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829540.034419:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790729984/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829540.034425:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829540.034427:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829540.034428:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790729984:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829540.034434:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790730112:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829540.034440:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790730368/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829540.034451:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829540.034455:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829540.034461:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829540/real 1787829540] req@ffff9e680a2d43c0 x1874668790730112/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829540.034462:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829540.034470:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829540.034479:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790730112/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829540.034484:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6176 00000100:00080000:2.0:1787829540.034495:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829540.034497:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829540.034501:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829540.034507:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790730112:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829540.034513:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829540.034522:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829540.034527:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790730240:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829540.034546:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790730496/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829540.034547:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829540.034559:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829540/real 1787829540] req@ffff9e680a2d4780 x1874668790730240/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829540.034583:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790730240/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829540.034590:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829540.034590:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829540.034597:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829540.034604:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790730240:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829540.034606:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829540.034611:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829540.034612:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790730368:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829540.034634:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6176 00000800:00000100:2.0:1787829540.034639:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829540.034640:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829540.034651:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829540/real 1787829540] req@ffff9e680a2d52c0 x1874668790730368/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829540.034651:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829540.034669:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829540.034672:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790730368/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829540.034685:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829540.034687:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829540.034690:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790730368:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829540.034721:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790730496:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829540.034733:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790730624/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829540.034740:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829540.034756:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829540/real 1787829540] req@ffff9e680a2d61c0 x1874668790730496/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829540.034757:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829540.034778:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829540.034782:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790730496/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829540.034783:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829540.034800:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829540.034800:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6176 00000100:00080000:2.0:1787829540.034812:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829540.034815:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829540.034816:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790730496:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829540.034829:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829540.034833:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790730624:192.168.203.109@tcp:8:kworker.0 00000008:00100000:3.0:1787829540.034840:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829540.034851:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829540.034861:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790730752/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829540.034863:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829540/real 1787829540] req@ffff9e680a2d6d00 x1874668790730624/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829540.034898:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790730624/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829540.034929:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829540.034933:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829540.034936:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790730624:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829540.034950:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790730752:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829540.034965:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829540.034972:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829540/real 1787829540] req@ffff9e680a2d5e00 x1874668790730752/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829540.034979:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790730752/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829540.034984:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829540.034986:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829540.034987:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790730752:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829540.545848:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829540.737925:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829541.569798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829541.761847:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829542.593895:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829542.785863:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829543.617836:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829543.809848:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829544.641849:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829544.833926:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829545.153796:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829545.153808:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829545.153813:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829545.153822:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6181 00000100:00080000:2.0:1787829545.153832:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829545.153838:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829545.153881:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790730880/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829545.153903:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829545.153907:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829545.153909:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829545.153913:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6181 00000100:00080000:2.0:1787829545.153918:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829545.153922:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829545.153932:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829545.153945:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790730880:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829545.153958:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790731008/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829545.153968:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829545.153972:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829545.153974:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829545.153977:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6181 00000100:00080000:2.0:1787829545.153980:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829545.153984:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829545.153988:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829545.153997:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790731136/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829545.154003:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829545.154006:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829545.154009:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829545.154011:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829545.154015:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6181 00000100:00080000:2.0:1787829545.154018:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829545.154022:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829545.154026:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829545.154035:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790731264/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829545.154040:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829545/real 1787829545] req@ffff9e6804423480 x1874668790730880/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829545.154045:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829545.154048:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829545.154050:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829545.154053:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6181 00000100:00100000:1.0:1787829545.154056:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790730880/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829545.154056:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829545.154060:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829545.154065:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829545.154074:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790731392/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829545.154078:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829545.154082:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829545.154083:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829545.154086:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787829545.154087:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790730880:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829545.154088:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829545.154091:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6181 00000100:00080000:2.0:1787829545.154094:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829545.154098:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829545.154102:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829545.154109:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790731008:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829545.154111:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790731520/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829545.154121:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829545.154124:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829545.154125:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829545.154129:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6181 00000100:00080000:2.0:1787829545.154132:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829545.154135:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1787829545.154139:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:2.0:1787829545.154149:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829545.154151:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829545/real 1787829545] req@ffff9e6804420f00 x1874668790731008/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829545.154163:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790731648/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829545.154172:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790731008/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829545.154173:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829545.154177:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829545.154178:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829545.154182:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6181 00000100:00080000:1.0:1787829545.154186:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829545.154186:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829545.154190:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829545.154190:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829545.154193:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790731008:192.168.203.109@tcp:38:kworker.0 00000008:00100000:2.0:1787829545.154194:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829545.154200:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790731136:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829545.154203:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790731776/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829545.154212:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829545.154216:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829545.154218:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829545.154219:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829545.154221:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6181 00000100:00080000:2.0:1787829545.154225:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829545.154229:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829545/real 1787829545] req@ffff9e6804422580 x1874668790731136/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829545.154229:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829545.154233:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829545.154241:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790731136/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829545.154249:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829545.154253:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829545.154256:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790731904/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829545.154257:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790731136:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829545.154269:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790731264:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829545.154294:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829545.154306:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829545/real 1787829545] req@ffff9e6804421680 x1874668790731264/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829545.154314:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790731264/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829545.154325:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829545.154327:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829545.154329:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790731264:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829545.154333:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790731392:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829545.154347:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829545.154353:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829545/real 1787829545] req@ffff9e6804422d00 x1874668790731392/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829545.154360:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790731392/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829545.154365:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829545.154367:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829545.154369:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790731392:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829545.154381:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790731520:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829545.154391:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829545.154398:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829545/real 1787829545] req@ffff9e6804422940 x1874668790731520/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829545.154405:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790731520/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829545.154411:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829545.154412:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829545.154415:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790731520:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829545.154419:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790731648:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829545.154428:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829545.154434:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829545/real 1787829545] req@ffff9e6804420780 x1874668790731648/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829545.154441:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790731648/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829545.154446:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829545.154447:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829545.154449:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790731648:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829545.154452:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790731776:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829545.154462:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829545.154467:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829545/real 1787829545] req@ffff9e68044221c0 x1874668790731776/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829545.154474:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790731776/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829545.154479:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829545.154481:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829545.154483:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790731776:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829545.154486:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790731904:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829545.154495:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829545.154501:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829545/real 1787829545] req@ffff9e6804423840 x1874668790731904/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829545.154508:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790731904/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829545.154513:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829545.154514:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829545.154516:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790731904:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829545.665777:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829545.857875:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829546.689783:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829546.881818:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829547.713803:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829547.905861:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829548.737791:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829548.929870:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829549.761812:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829549.953889:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829550.273805:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829550.273813:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829550.273817:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829550.273825:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6186 00000100:00080000:0.0:1787829550.273845:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829550.273851:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829550.273885:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790732032/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829550.273920:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829550.273924:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829550.273926:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829550.273931:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6186 00000100:00080000:0.0:1787829550.273935:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829550.273939:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829550.273948:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829550.273951:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790732032:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787829550.273961:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790732160/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829550.273970:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829550.273973:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829550.273974:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829550.273977:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6186 00000100:00080000:0.0:1787829550.273980:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829550.273982:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829550.273986:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829550.273993:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790732288/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829550.274000:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829550.274002:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829550.274004:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829550.274008:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829550.274012:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6186 00000100:00080000:0.0:1787829550.274016:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829550.274019:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829550.274023:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829550.274043:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790732416/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829550.274047:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829550/real 1787829550] req@ffff9e6803e9ad00 x1874668790732032/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829550.274074:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829550.274078:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790732032/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829550.274081:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829550.274085:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829550.274094:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6186 00000100:00080000:1.0:1787829550.274094:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829550.274102:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829550.274102:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829550.274108:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790732032:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829550.274110:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829550.274118:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829550.274133:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790732160:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829550.274134:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790732544/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829550.274150:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829550.274155:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829550.274159:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829550.274163:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829550.274167:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829550/real 1787829550] req@ffff9e6803e98f00 x1874668790732160/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829550.274170:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6186 00000100:00080000:0.0:1787829550.274177:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829550.274183:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790732160/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829550.274184:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829550.274192:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829550.274195:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829550.274200:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829550.274203:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790732160:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829550.274206:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790732672/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829550.274212:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790732288:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829550.274228:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829550.274233:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829550.274234:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829550.274238:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829550.274243:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6186 00000100:00100000:1.0:1787829550.274245:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829550/real 1787829550] req@ffff9e6803e992c0 x1874668790732288/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829550.274247:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829550.274254:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829550.274261:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829550.274262:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790732288/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829550.274269:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829550.274270:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790732800/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829550.274274:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829550.274284:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790732288:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829550.274285:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829550.274293:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829550.274296:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829550.274298:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790732416:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787829550.274304:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6186 00000100:00080000:0.0:1787829550.274310:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829550.274315:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829550.274320:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829550.274327:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829550.274336:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790732928/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829550.274342:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829550/real 1787829550] req@ffff9e6803e9bc00 x1874668790732416/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829550.274350:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829550.274355:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829550.274357:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829550.274358:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790732416/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829550.274365:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6186 00000100:00080000:0.0:1787829550.274372:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829550.274372:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829550.274380:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829550.274381:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829550.274386:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790732416:192.168.203.109@tcp:8:kworker.0 00000008:00100000:0.0:1787829550.274388:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829550.274400:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790733056/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829550.274405:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790732544:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829550.274426:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829550.274434:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829550/real 1787829550] req@ffff9e6803e9a940 x1874668790732544/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829550.274442:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790732544/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829550.274447:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829550.274449:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829550.274451:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790732544:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829550.274455:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790732672:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829550.274464:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829550.274470:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829550/real 1787829550] req@ffff9e6803e98b40 x1874668790732672/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829550.274477:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790732672/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829550.274483:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829550.274485:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829550.274486:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790732672:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829550.274490:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790732800:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829550.274499:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829550.274505:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829550/real 1787829550] req@ffff9e6803e98780 x1874668790732800/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829550.274512:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790732800/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829550.274517:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829550.274518:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829550.274520:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790732800:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829550.274523:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790732928:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829550.274533:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829550.274538:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829550/real 1787829550] req@ffff9e6803e9a1c0 x1874668790732928/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829550.274545:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790732928/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829550.274549:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829550.274552:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829550.274553:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790732928:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829550.274562:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790733056:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829550.274572:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829550.274577:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829550/real 1787829550] req@ffff9e6803e9b840 x1874668790733056/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829550.274584:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790733056/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829550.274589:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829550.274592:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829550.274594:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790733056:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829550.785775:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829550.977868:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829551.809766:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829552.001856:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829552.833773:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829553.025880:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829553.857816:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829554.049858:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829554.881777:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829555.073864:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829555.393784:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829555.393793:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829555.393809:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829555.393818:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6192 00000100:00080000:0.0:1787829555.393827:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829555.393833:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829555.393865:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790733184/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829555.393887:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829555.393890:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829555.393891:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829555.393894:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6192 00000100:00080000:0.0:1787829555.393896:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829555.393899:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829555.393907:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829555.393917:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790733312/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829555.393919:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790733184:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829555.393933:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829555.393938:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829555.393939:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829555.393943:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6192 00000100:00080000:0.0:1787829555.393945:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829555.393947:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829555.393951:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829555.393958:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790733440/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829555.393965:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829555.393967:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829555.393969:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829555.393971:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6192 00000100:00080000:0.0:1787829555.393973:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829555.393975:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829555.393978:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829555.393983:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829555.393989:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790733568/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829555.393997:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829555.393999:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829555.394000:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829555.394003:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6192 00000100:00080000:0.0:1787829555.394005:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829555.394007:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829555.394011:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829555.394020:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829555/real 1787829555] req@ffff9e6803e9a580 x1874668790733184/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829555.394021:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790733696/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829555.394054:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829555.394057:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829555.394058:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829555.394063:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6192 00000100:00100000:1.0:1787829555.394064:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790733184/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829555.394065:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829555.394072:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829555.394078:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829555.394084:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829555.394089:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790733824/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829555.394092:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829555.394099:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790733184:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829555.394102:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829555.394107:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829555.394108:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829555.394111:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6192 00000100:00080000:0.0:1787829555.394113:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829555.394116:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829555.394118:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829555.394124:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790733312:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829555.394127:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790733952/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829555.394136:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829555.394139:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829555.394141:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829555.394144:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6192 00000100:00080000:0.0:1787829555.394146:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829555.394150:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829555.394153:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829555.394158:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829555.394167:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829555/real 1787829555] req@ffff9e6803e99680 x1874668790733312/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829555.394168:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790734080/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829555.394203:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829555.394206:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829555.394208:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790733312/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829555.394209:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829555.394217:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6192 00000100:00080000:0.0:1787829555.394222:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829555.394226:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787829555.394227:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787829555.394229:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829555.394232:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829555.394238:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790733312:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829555.394240:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790734208/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829555.394255:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790733440:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829555.394274:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829555.394282:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829555/real 1787829555] req@ffff9e6803e99e00 x1874668790733440/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829555.394290:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790733440/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829555.394295:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829555.394297:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829555.394299:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790733440:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829555.394302:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790733568:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829555.394313:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829555.394320:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829555/real 1787829555] req@ffff9e6803e9b480 x1874668790733568/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829555.394328:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790733568/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829555.394333:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829555.394335:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829555.394339:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790733568:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829555.394343:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790733696:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829555.394357:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829555.394364:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829555/real 1787829555] req@ffff9e6803e9b840 x1874668790733696/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829555.394371:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790733696/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829555.394377:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829555.394378:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829555.394380:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790733696:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829555.394383:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790733824:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829555.394395:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829555.394402:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829555/real 1787829555] req@ffff9e6803e9a1c0 x1874668790733824/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829555.394409:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790733824/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829555.394416:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829555.394418:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829555.394421:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790733824:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829555.394438:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790733952:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829555.394452:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829555.394460:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829555/real 1787829555] req@ffff9e6803e98780 x1874668790733952/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829555.394467:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790733952/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829555.394473:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829555.394476:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829555.394478:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790733952:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829555.394482:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790734080:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829555.394494:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829555.394500:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829555/real 1787829555] req@ffff9e6803e98b40 x1874668790734080/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829555.394507:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790734080/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829555.394513:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829555.394516:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829555.394518:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790734080:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829555.394523:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790734208:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829555.394540:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829555.394552:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829555/real 1787829555] req@ffff9e6803e9a940 x1874668790734208/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829555.394566:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790734208/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829555.394576:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829555.394580:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829555.394585:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790734208:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829555.905824:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829556.097935:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829556.929822:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829557.121883:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829557.953870:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829558.145905:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829558.977862:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829559.169897:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829560.001867:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829560.193916:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829560.513834:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829560.513846:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829560.513850:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829560.513861:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6197 00000100:00080000:2.0:1787829560.513874:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829560.513881:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829560.513923:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790734336/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829560.513955:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829560.513958:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829560.513960:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829560.513963:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6197 00000100:00080000:2.0:1787829560.513966:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829560.513968:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829560.513984:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829560.513995:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790734464/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829560.514005:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829560.514008:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829560.514011:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829560.514017:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790734336:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829560.514017:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6197 00000100:00080000:2.0:1787829560.514029:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829560.514034:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829560.514038:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829560.514049:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790734592/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829560.514057:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829560.514060:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829560.514061:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829560.514064:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6197 00000100:00080000:2.0:1787829560.514070:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829560.514073:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829560.514078:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829560.514087:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790734720/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829560.514098:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829560.514111:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829560.514115:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829560.514119:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829560.514123:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6197 00000100:00080000:2.0:1787829560.514126:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829560.514130:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829560.514134:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829560.514147:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790734848/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829560.514149:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829560/real 1787829560] req@ffff9e68044212c0 x1874668790734336/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829560.514206:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829560.514210:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829560.514212:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829560.514217:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6197 00000100:00080000:2.0:1787829560.514220:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829560.514222:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790734336/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829560.514227:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829560.514235:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829560.514245:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790734976/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829560.514271:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829560.514281:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829560.514284:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829560.514290:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829560.514293:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829560.514296:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790734336:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829560.514301:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6197 00000100:00080000:2.0:1787829560.514308:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829560.514312:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829560.514317:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829560.514329:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790735104/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829560.514340:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829560.514341:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790734464:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829560.514350:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829560.514358:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829560.514366:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6197 00000100:00080000:2.0:1787829560.514369:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829560.514372:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829560.514376:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829560.514385:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790735232/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829560.514396:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829560.514409:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829560.514413:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829560.514415:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829560.514422:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6197 00000100:00100000:1.0:1787829560.514423:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829560/real 1787829560] req@ffff9e68044230c0 x1874668790734464/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829560.514430:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829560.514444:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829560.514451:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829560.514456:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790734464/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829560.514467:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787829560.514467:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790735360/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829560.514479:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829560.514497:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790734464:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829560.514511:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790734592:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829560.514534:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829560.514541:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829560/real 1787829560] req@ffff9e6804420b40 x1874668790734592/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829560.514549:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790734592/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829560.514554:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829560.514556:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829560.514558:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790734592:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829560.514562:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790734720:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829560.514573:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829560.514578:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829560/real 1787829560] req@ffff9e6804421e00 x1874668790734720/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829560.514585:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790734720/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829560.514590:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829560.514592:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829560.514594:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790734720:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829560.514598:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790734848:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829560.514608:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829560.514614:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829560/real 1787829560] req@ffff9e6804423840 x1874668790734848/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829560.514626:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790734848/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829560.514631:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829560.514633:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829560.514635:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790734848:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829560.514638:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790734976:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829560.514648:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829560.514654:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829560/real 1787829560] req@ffff9e68044221c0 x1874668790734976/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829560.514662:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790734976/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829560.514668:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829560.514670:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829560.514671:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790734976:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829560.514685:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790735104:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829560.514720:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829560.514729:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829560/real 1787829560] req@ffff9e6804420780 x1874668790735104/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829560.514737:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790735104/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829560.514746:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829560.514748:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829560.514750:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790735104:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829560.514754:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790735232:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829560.514783:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829560.514791:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829560/real 1787829560] req@ffff9e6804422940 x1874668790735232/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829560.514799:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790735232/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829560.514804:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829560.514806:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829560.514808:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790735232:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829560.514812:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790735360:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829560.514823:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829560.514828:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829560/real 1787829560] req@ffff9e6804422d00 x1874668790735360/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829560.514835:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790735360/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829560.514840:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829560.514842:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829560.514844:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790735360:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829561.025829:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829561.217904:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829562.049862:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829562.241897:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829563.073834:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829563.265884:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829564.097836:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829564.289887:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829565.121842:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829565.313920:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829565.633817:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829565.633828:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829565.633833:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829565.633843:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6202 00000100:00080000:1.0:1787829565.633855:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829565.633862:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829565.633900:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790735488/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829565.633935:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829565.633940:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829565.633942:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829565.633947:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6202 00000100:00080000:1.0:1787829565.633951:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829565.633957:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829565.633968:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829565.633985:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790735616/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829565.633985:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790735488:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829565.634011:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829565.634015:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829565.634018:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829565.634025:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6202 00000100:00080000:1.0:1787829565.634030:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829565.634035:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829565.634041:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829565.634053:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790735744/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829565.634064:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829565.634068:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829565.634070:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829565.634075:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6202 00000100:00080000:1.0:1787829565.634080:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829565.634084:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829565.634089:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829565.634103:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790735872/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829565.634113:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829565.634117:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829565.634119:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829565.634123:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6202 00000100:00080000:1.0:1787829565.634127:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829565.634131:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829565.634142:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829565.634147:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790735616:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829565.634164:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790736000/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829565.634185:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829565.634189:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829565.634193:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829565.634194:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790735744:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829565.634208:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6202 00000100:00080000:1.0:1787829565.634217:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829565.634223:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829565.634230:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829565.634252:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790736128/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829565.634253:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790735872:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787829565.634273:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829565.634278:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829565.634281:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829565.634288:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6202 00000100:00080000:1.0:1787829565.634294:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829565.634298:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790736000:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787829565.634304:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829565.634312:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829565.634325:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790736256/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829565.634327:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790736128:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829565.634337:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829565.634345:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829565.634350:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829565.634354:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6202 00000100:00080000:1.0:1787829565.634360:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829565.634363:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829565.634366:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829565.634375:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790736384/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829565.634384:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829565.634387:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829565.634389:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829565.634392:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6202 00000100:00100000:2.0:1787829565.634397:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790736256:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829565.634398:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829565.634407:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829565.634414:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829565.634431:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790736512/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829565.634444:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790736384:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829565.634488:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790736512:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829566.145809:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829566.337934:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829567.169830:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829567.361865:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829568.193793:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829568.385880:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787829568.705987:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:3.0:1787829568.706035:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787829568.706053:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787829568.706114:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:3.0:1787829568.706130:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787829568.706142:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787829568.706151:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787829568.706159:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787829568.706169:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787829568.706177:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:3.0:1787829568.706185:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787829568.706388:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829565/real 1787829568] req@ffff9e6831aa9680 x1874668790736512/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829568.706426:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790736512/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829568.706438:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829568.706443:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829568.706447:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790736512:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829568.706456:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829565/real 1787829568] req@ffff9e6831aabc00 x1874668790736256/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829568.706470:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790736256/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829568.706478:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829568.706482:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829568.706486:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790736256:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829568.706493:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829565/real 1787829568] req@ffff9e6831aa9a40 x1874668790736384/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829568.706505:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790736384/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829568.706513:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829568.706517:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829568.706520:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790736384:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829568.706529:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829565/real 1787829568] req@ffff9e6831aa8000 x1874668790736000/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829568.706542:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790736000/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829568.706549:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829568.706552:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829568.706555:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790736000:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829568.706562:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829565/real 1787829568] req@ffff9e6831aaa940 x1874668790736128/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829568.706574:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790736128/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829568.706584:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829568.706587:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829568.706589:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790736128:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829568.706596:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829565/real 1787829568] req@ffff9e6831aa8780 x1874668790735616/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829568.706609:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790735616/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829568.706617:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829568.706620:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829568.706623:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790735616:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829568.706629:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829565/real 1787829568] req@ffff9e6831aaa1c0 x1874668790735744/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829568.706639:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790735744/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829568.706646:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829568.706650:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829568.706653:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790735744:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829568.706659:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829565/real 1787829568] req@ffff9e6831aaad00 x1874668790735872/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829568.706669:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790735872/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829568.706677:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829568.706680:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829568.706683:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790735872:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829568.706690:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829565/real 1787829568] req@ffff9e6831aab480 x1874668790735488/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829568.706761:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790735488/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829568.706771:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829568.706775:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829568.706779:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790735488:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829569.217863:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829569.409877:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829570.241814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829570.433935:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829570.753827:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829570.753841:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829570.753846:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829570.753860:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6207 00000100:00080000:0.0:1787829570.753874:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829570.753884:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829570.753931:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790736640/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829570.753974:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829570.753978:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829570.753980:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829570.753985:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6207 00000100:00080000:0.0:1787829570.753989:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829570.753994:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829570.754006:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829570.754024:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790736768/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829570.754039:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829570.754044:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829570.754047:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829570.754057:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6207 00000100:00100000:2.0:1787829570.754058:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790736640:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829570.754063:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829570.754076:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829570.754084:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829570.754106:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790736896/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829570.754115:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829570.754120:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829570.754122:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829570.754127:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6207 00000100:00080000:0.0:1787829570.754136:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829570.754142:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829570.754149:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829570.754152:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829570.754173:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790737024/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829570.754203:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829570.754207:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829570.754212:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829570.754214:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829570/real 1787829570] req@ffff9e6803e9bc00 x1874668790736640/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829570.754223:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6207 00000100:00080000:0.0:1787829570.754236:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829570.754248:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829570.754256:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829570.754257:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790736640/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829570.754276:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829570.754281:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790737152/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829570.754301:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829570.754309:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829570.754316:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790736640:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829570.754319:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829570.754322:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829570.754327:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6207 00000100:00080000:0.0:1787829570.754332:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829570.754338:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829570.754347:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829570.754367:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790737280/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829570.754386:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790736768:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829570.754392:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829570.754398:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829570.754400:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829570.754406:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6207 00000100:00080000:0.0:1787829570.754413:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829570.754418:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829570.754425:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829570.754428:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829570.754452:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790737408/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829570.754461:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829570/real 1787829570] req@ffff9e6803e992c0 x1874668790736768/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829570.754474:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829570.754479:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829570.754482:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829570.754495:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6207 00000100:00080000:0.0:1787829570.754503:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829570.754505:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790736768/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829570.754517:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829570.754531:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829570.754536:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829570.754544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829570.754551:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790737536/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829570.754553:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790736768:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829570.754581:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790736896:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829570.754583:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829570.754595:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829570.754602:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829570.754609:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6207 00000100:00080000:0.0:1787829570.754615:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829570.754621:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829570.754627:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829570.754636:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829570.754656:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790737664/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829570.754658:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829570/real 1787829570] req@ffff9e6803e98f00 x1874668790736896/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829570.754783:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790736896/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829570.754793:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829570.754796:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829570.754800:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790736896:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829570.754808:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790737024:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829570.754834:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829570.754845:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829570/real 1787829570] req@ffff9e6803e9ad00 x1874668790737024/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829570.754857:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790737024/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829570.754866:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829570.754870:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829570.754873:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790737024:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829570.754880:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790737152:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829570.754907:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829570.754923:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829570/real 1787829570] req@ffff9e6803e9a940 x1874668790737152/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829570.754947:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790737152/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829570.754956:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829570.754960:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829570.754964:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790737152:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829570.754986:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790737280:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829570.755008:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829570.755029:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829570/real 1787829570] req@ffff9e6803e98b40 x1874668790737280/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829570.755042:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790737280/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829570.755052:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829570.755055:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829570.755058:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790737280:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829570.755065:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790737408:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829570.755086:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829570.755097:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829570/real 1787829570] req@ffff9e6803e98780 x1874668790737408/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829570.755109:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790737408/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829570.755123:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829570.755127:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829570.755131:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790737408:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829570.755137:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790737536:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829570.755156:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829570.755168:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829570/real 1787829570] req@ffff9e6803e9a1c0 x1874668790737536/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829570.755179:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790737536/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829570.755186:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829570.755190:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829570.755193:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790737536:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829570.755200:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790737664:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829570.755221:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829570.755231:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829570/real 1787829570] req@ffff9e6803e9b840 x1874668790737664/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829570.755243:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790737664/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829570.755250:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829570.755253:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829570.755257:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790737664:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829571.265772:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829571.457876:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829572.289819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829572.481888:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829573.313831:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829573.505890:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829574.337829:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829574.529869:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829575.361833:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829575.553920:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829575.873825:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829575.873836:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829575.873842:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829575.873853:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6212 00000100:00080000:0.0:1787829575.873865:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829575.873873:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829575.873919:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790737792/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829575.873946:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829575.873949:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829575.873950:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829575.873953:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6212 00000100:00080000:0.0:1787829575.873956:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829575.873959:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829575.873968:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829575.873977:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790737920/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829575.873984:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829575.873989:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829575.873990:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829575.873993:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6212 00000100:00080000:0.0:1787829575.873995:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829575.873997:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829575.874005:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829575.874011:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790738048/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829575.874019:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829575.874022:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829575.874023:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829575.874028:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6212 00000100:00080000:0.0:1787829575.874033:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829575.874035:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829575.874039:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829575.874046:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790738176/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829575.874055:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829575.874060:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829575.874063:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829575.874065:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790737792:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829575.874069:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6212 00000100:00080000:0.0:1787829575.874077:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829575.874082:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829575.874087:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829575.874104:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790738304/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829575.874111:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829575.874114:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829575.874115:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829575.874119:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6212 00000100:00080000:0.0:1787829575.874123:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829575.874126:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829575.874130:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829575.874138:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790738432/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829575.874150:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829575.874161:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829575.874167:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829575.874169:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829575.874175:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6212 00000100:00080000:0.0:1787829575.874182:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829575.874187:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829575.874194:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829575.874200:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829575/real 1787829575] req@ffff9e6803e9b480 x1874668790737792/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829575.874214:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790738560/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829575.874226:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790737792/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829575.874228:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829575.874240:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829575.874246:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829575.874259:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6212 00000100:00080000:2.0:1787829575.874264:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829575.874265:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829575.874280:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787829575.874280:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829575.874288:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790737792:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1787829575.874293:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829575.874299:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790737920:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829575.874314:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790738688/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829575.874327:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829575.874328:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829575.874334:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829575.874337:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829575.874342:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6212 00000100:00080000:0.0:1787829575.874347:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829575.874349:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829575/real 1787829575] req@ffff9e6803e99e00 x1874668790737920/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829575.874354:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829575.874362:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829575.874382:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790738816/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829575.874388:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790737920/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829575.874425:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829575.874429:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829575.874438:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790737920:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829575.874445:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790738048:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829575.874463:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829575.874471:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829575/real 1787829575] req@ffff9e6803e99680 x1874668790738048/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829575.874479:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790738048/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829575.874484:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829575.874486:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829575.874488:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790738048:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829575.874508:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790738176:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829575.874519:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829575.874525:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829575/real 1787829575] req@ffff9e6803e9a580 x1874668790738176/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829575.874532:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790738176/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829575.874537:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829575.874539:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829575.874541:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790738176:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829575.874545:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790738304:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829575.874559:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829575.874566:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829575/real 1787829575] req@ffff9e6803e9b840 x1874668790738304/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829575.874574:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790738304/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829575.874580:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829575.874582:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829575.874585:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790738304:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829575.874589:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790738432:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829575.874598:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829575.874604:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829575/real 1787829575] req@ffff9e6803e9a1c0 x1874668790738432/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829575.874611:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790738432/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829575.874617:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829575.874619:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829575.874621:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790738432:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829575.874624:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790738560:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829575.874634:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829575.874639:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829575/real 1787829575] req@ffff9e6803e98780 x1874668790738560/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829575.874646:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790738560/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829575.874673:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829575.874677:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829575.874680:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790738560:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829575.874686:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790738688:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829575.874733:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829575.874741:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829575/real 1787829575] req@ffff9e6803e98b40 x1874668790738688/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829575.874749:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790738688/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829575.874754:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829575.874757:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829575.874759:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790738688:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829575.874762:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790738816:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829575.874773:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829575.874779:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829575/real 1787829575] req@ffff9e6803e9a940 x1874668790738816/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829575.874786:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790738816/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829600 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829575.874790:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829575.874792:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829575.874795:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790738816:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829576.385825:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829576.577869:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829577.409836:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829577.601910:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829578.433798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829578.625838:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829579.457832:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829579.649861:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829580.481797:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829580.673872:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829580.993817:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829580.993830:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829580.993835:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829580.993846:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6217 00000100:00080000:1.0:1787829580.993860:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829580.993868:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829580.993915:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790738944/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829580.993941:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829580.993944:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829580.993946:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829580.993949:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6217 00000100:00080000:1.0:1787829580.993952:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829580.993954:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829580.993965:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829580.993975:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790739072/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829580.993983:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829580.993985:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829580.993987:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829580.993991:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6217 00000100:00100000:2.0:1787829580.993993:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790738944:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829580.993994:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829580.994000:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829580.994005:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829580.994011:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790739200/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829580.994023:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829580.994026:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829580.994027:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829580.994030:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6217 00000100:00080000:1.0:1787829580.994033:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829580.994037:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829580.994041:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829580.994050:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790739328/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829580.994056:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829580.994059:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829580.994060:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829580.994063:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6217 00000100:00080000:1.0:1787829580.994067:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829580.994069:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829580.994072:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829580.994080:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829580.994082:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790739456/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829580.994090:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829580.994093:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829580.994095:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829580.994098:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6217 00000100:00080000:1.0:1787829580.994106:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829580.994109:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829580.994112:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829580.994120:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790739584/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829580.994132:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829580/real 1787829580] req@ffff9e6831aa8b40 x1874668790738944/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829580.994134:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829580.994141:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829580.994143:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829580.994150:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6217 00000100:00080000:1.0:1787829580.994153:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829580.994156:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829580.994159:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829580.994162:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790738944/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829580.994169:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790739712/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829580.994177:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829580.994179:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829580.994184:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829580.994187:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829580.994188:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829580.994195:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6217 00000100:00100000:2.0:1787829580.994198:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790738944:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829580.994199:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829580.994205:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829580.994209:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829580.994217:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790739840/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829580.994224:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829580.994226:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829580.994228:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829580.994230:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6217 00000100:00080000:1.0:1787829580.994233:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829580.994236:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790739072:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829580.994239:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829580.994246:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829580.994255:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790739968/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829580.994266:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829580.994279:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829580/real 1787829580] req@ffff9e6831aa9e00 x1874668790739072/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829580.994288:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790739072/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829580.994295:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829580.994297:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829580.994300:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790739072:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829580.994305:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790739200:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829580.994324:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829580.994330:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829580/real 1787829580] req@ffff9e6831aa8f00 x1874668790739200/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829580.994338:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790739200/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829580.994343:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829580.994345:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829580.994347:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790739200:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829580.994351:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790739328:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829580.994362:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829580.994374:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829580/real 1787829580] req@ffff9e6831aaa580 x1874668790739328/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829580.994385:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790739328/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829580.994391:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829580.994394:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829580.994396:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790739328:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829580.994400:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790739456:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829580.994412:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829580.994419:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829580/real 1787829580] req@ffff9e6831aab480 x1874668790739456/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829580.994426:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790739456/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829580.994431:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829580.994433:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829580.994435:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790739456:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829580.994438:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790739584:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829580.994448:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829580.994454:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829580/real 1787829580] req@ffff9e6831aaad00 x1874668790739584/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829580.994462:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790739584/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829580.994469:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829580.994470:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829580.994472:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790739584:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829580.994476:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790739712:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829580.994486:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829580.994491:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829580/real 1787829580] req@ffff9e6831aaa1c0 x1874668790739712/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829580.994499:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790739712/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829580.994504:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829580.994507:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829580.994509:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790739712:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829580.994512:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790739840:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829580.994522:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829580.994527:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829580/real 1787829580] req@ffff9e6831aa8780 x1874668790739840/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829580.994534:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790739840/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829580.994539:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829580.994541:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829580.994543:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790739840:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829580.994556:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790739968:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829580.994566:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829580.994572:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829580/real 1787829580] req@ffff9e6831aaa940 x1874668790739968/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829580.994579:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790739968/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829605 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829580.994584:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829580.994586:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829580.994588:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790739968:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829581.505790:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829581.697867:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829582.529796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829582.721904:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829583.553800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829583.745875:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829584.577790:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829584.769871:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829585.601813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829585.793860:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829586.113810:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829586.113821:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829586.113826:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829586.113837:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6222 00000100:00080000:2.0:1787829586.113850:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829586.113858:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829586.113909:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790740096/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829586.113940:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829586.113943:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829586.113944:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829586.113949:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6222 00000100:00080000:2.0:1787829586.113953:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829586.113958:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829586.113970:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829586.113986:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790740224/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829586.113999:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829586.114004:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787829586.114010:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790740096:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829586.114010:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829586.114019:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6222 00000100:00080000:2.0:1787829586.114024:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829586.114028:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829586.114034:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829586.114047:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790740352/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829586.114055:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829586.114058:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829586.114059:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829586.114062:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6222 00000100:00080000:2.0:1787829586.114066:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829586.114068:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829586.114072:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829586.114080:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790740480/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829586.114087:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829586.114089:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829586.114090:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829586.114092:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6222 00000100:00080000:2.0:1787829586.114095:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829586.114097:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829586.114101:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829586.114105:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829586.114115:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790740608/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829586.114122:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829586.114125:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829586.114126:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829586.114128:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6222 00000100:00080000:2.0:1787829586.114131:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829586.114134:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829586.114138:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829586.114145:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829586/real 1787829586] req@ffff9e6804421680 x1874668790740096/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829586.114147:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790740736/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829586.114156:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829586.114159:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829586.114162:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829586.114171:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6222 00000100:00080000:2.0:1787829586.114175:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829586.114179:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790740096/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829586.114183:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829586.114194:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829586.114196:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829586.114202:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829586.114206:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790740096:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829586.114212:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790740864/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829586.114222:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829586.114224:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829586.114226:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829586.114230:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6222 00000100:00100000:1.0:1787829586.114234:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790740224:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829586.114236:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829586.114243:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829586.114248:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829586.114260:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790740992/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829586.114265:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829586.114275:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829586.114278:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829586.114279:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829586.114284:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6222 00000100:00080000:2.0:1787829586.114288:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829586.114289:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829586/real 1787829586] req@ffff9e6804422580 x1874668790740224/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829586.114297:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829586.114306:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790740224/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829586.114306:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829586.114326:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790741120/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829586.114327:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829586.114337:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829586.114346:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790740224:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829586.114355:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790740352:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829586.114379:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829586.114389:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829586/real 1787829586] req@ffff9e6804420f00 x1874668790740352/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829586.114397:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790740352/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829586.114403:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829586.114405:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829586.114407:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790740352:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829586.114411:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790740480:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829586.114424:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829586.114430:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829586/real 1787829586] req@ffff9e6804423480 x1874668790740480/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829586.114438:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790740480/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829586.114444:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829586.114445:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829586.114448:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790740480:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829586.114452:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790740608:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829586.114463:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829586.114469:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829586/real 1787829586] req@ffff9e6804422d00 x1874668790740608/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829586.114475:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790740608/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829586.114481:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829586.114482:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829586.114484:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790740608:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829586.114488:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790740736:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829586.114498:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829586.114503:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829586/real 1787829586] req@ffff9e6804422940 x1874668790740736/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829586.114510:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790740736/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829586.114516:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829586.114518:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829586.114520:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790740736:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829586.114523:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790740864:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829586.114533:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829586.114539:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829586/real 1787829586] req@ffff9e6804420780 x1874668790740864/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829586.114546:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790740864/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829586.114551:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829586.114553:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829586.114555:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790740864:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829586.114568:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790740992:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829586.114579:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829586.114605:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829586/real 1787829586] req@ffff9e68044221c0 x1874668790740992/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829586.114613:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790740992/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829586.114618:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829586.114621:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829586.114623:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790740992:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829586.114626:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790741120:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829586.114645:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829586.114652:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829586/real 1787829586] req@ffff9e6804423840 x1874668790741120/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829586.114659:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790741120/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829586.114665:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829586.114667:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829586.114669:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790741120:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829586.625830:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829586.817858:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829587.649811:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829587.841931:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829588.673831:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829588.866411:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829589.697794:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829589.889892:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829590.721812:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829590.913904:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829591.233819:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829591.233834:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829591.233841:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829591.233855:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6227 00000100:00080000:2.0:1787829591.233871:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829591.233881:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829591.233931:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790741248/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829591.233973:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829591.233978:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829591.233981:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829591.233991:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6227 00000100:00080000:2.0:1787829591.233997:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829591.234005:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829591.234020:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829591.234036:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790741248:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829591.234041:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790741376/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829591.234071:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829591.234076:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829591.234080:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829591.234088:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6227 00000100:00080000:2.0:1787829591.234095:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829591.234100:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829591.234107:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829591.234128:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829591.234132:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790741504/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829591.234152:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829591.234157:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829591.234160:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829591.234167:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6227 00000100:00080000:2.0:1787829591.234174:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829591.234178:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829591/real 1787829591] req@ffff9e6804421e00 x1874668790741248/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829591.234186:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829591.234206:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829591.234211:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790741248/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829591.234227:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787829591.234234:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790741632/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829591.234235:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829591.234249:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790741248:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829591.234279:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829591.234284:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829591.234288:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829591.234294:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6227 00000100:00080000:2.0:1787829591.234301:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829591.234302:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790741376:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829591.234315:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829591.234325:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829591.234342:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829591.234348:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790741760/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829591.234364:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829591.234368:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829591/real 1787829591] req@ffff9e6804420b40 x1874668790741376/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829591.234370:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829591.234375:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829591.234389:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6227 00000100:00080000:2.0:1787829591.234399:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829591.234407:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829591.234412:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790741376/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829591.234417:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829591.234423:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829591.234429:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829591.234445:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790741376:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829591.234446:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790741888/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829591.234457:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790741504:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829591.234474:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829591.234478:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829591.234481:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829591.234490:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829591.234492:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6227 00000100:00080000:2.0:1787829591.234502:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829591.234506:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829591/real 1787829591] req@ffff9e68044230c0 x1874668790741504/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829591.234512:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829591.234529:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829591.234531:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790741504/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829591.234543:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829591.234561:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829591.234574:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790741504:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829591.234576:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790742016/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829591.234586:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790741632:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787829591.234606:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829591.234611:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829591.234613:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829591.234619:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829591.234621:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6227 00000100:00080000:2.0:1787829591.234630:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829591.234640:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829591.234641:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829591/real 1787829591] req@ffff9e68044212c0 x1874668790741632/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829591.234660:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829591.234663:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790741632/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829591.234673:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829591.234676:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829591.234687:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790741632:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829591.234689:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790742144/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829591.234741:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790741760:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787829591.234743:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829591.234754:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829591.234756:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829591.234765:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6227 00000800:00000100:1.0:1787829591.234771:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829591.234774:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829591.234786:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829591.234787:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829591/real 1787829591] req@ffff9e6804421e00 x1874668790741760/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829591.234805:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829591.234809:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790741760/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829591.234822:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829591.234825:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829591.234829:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790741760:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829591.234835:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790742272/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829591.234846:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790741888:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829591.234882:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829591.234897:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829591/real 1787829591] req@ffff9e6804423840 x1874668790741888/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829591.234910:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790741888/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829591.234920:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829591.234923:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829591.234925:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790741888:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829591.234933:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790742016:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829591.234952:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829591.234964:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829591/real 1787829591] req@ffff9e68044221c0 x1874668790742016/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829591.234974:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790742016/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829591.234982:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829591.234984:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829591.234987:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790742016:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829591.235003:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790742144:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829591.235024:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829591.235033:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829591/real 1787829591] req@ffff9e6804420780 x1874668790742144/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829591.235046:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790742144/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829591.235054:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829591.235057:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829591.235060:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790742144:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829591.235065:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790742272:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829591.235081:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829591.235088:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829591/real 1787829591] req@ffff9e6804422940 x1874668790742272/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829591.235095:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790742272/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829591.235100:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829591.235102:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829591.235105:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790742272:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829591.745839:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829591.937887:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829592.769833:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829592.961852:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829593.793807:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829593.985869:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829594.817827:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829595.009875:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829595.841817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829596.033863:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829596.353837:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829596.353848:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829596.353852:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829596.353863:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6233 00000100:00080000:2.0:1787829596.353875:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829596.353881:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829596.353923:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790742400/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829596.353951:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829596.353954:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829596.353955:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829596.353958:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6233 00000100:00080000:2.0:1787829596.353960:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829596.353962:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829596.353971:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829596.353980:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790742528/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829596.353986:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829596.353988:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829596.353989:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829596.353991:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6233 00000100:00080000:2.0:1787829596.353992:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829596.353994:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829596.353997:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829596.354001:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790742656/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829596.354007:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829596.354008:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829596.354009:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829596.354013:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6233 00000100:00080000:2.0:1787829596.354016:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829596.354019:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829596.354024:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829596.354035:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790742784/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829596.354042:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790742400:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829596.354050:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829596.354053:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829596.354056:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829596.354059:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6233 00000100:00080000:2.0:1787829596.354062:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829596.354064:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829596.354069:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829596.354077:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790742912/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829596.354083:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829596.354086:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829596.354087:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829596.354089:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6233 00000100:00080000:2.0:1787829596.354092:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829596.354094:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829596.354097:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829596.354104:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790743040/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829596.354109:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829596.354111:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829596.354112:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829596.354115:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6233 00000100:00080000:2.0:1787829596.354117:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829596.354118:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829596.354122:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829596.354126:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829596.354134:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790743168/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829596.354139:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829596.354142:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829596.354143:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829596.354146:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6233 00000100:00080000:2.0:1787829596.354148:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829596.354151:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829596.354153:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829596.354161:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790743296/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829596.354168:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829596/real 1787829596] req@ffff9e6804422d00 x1874668790742400/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829596.354168:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829596.354175:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829596.354177:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829596.354185:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6233 00000100:00080000:2.0:1787829596.354190:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829596.354192:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829596.354195:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829596.354202:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790742400/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829596.354203:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790743424/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829596.354236:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829596.354241:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829596.354246:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790742400:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787829596.354252:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790742528:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829596.354271:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829596.354279:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829596/real 1787829596] req@ffff9e6804423480 x1874668790742528/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829596.354287:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790742528/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829596.354301:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829596.354303:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829596.354305:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790742528:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829596.354309:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790742656:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829596.354322:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829596.354328:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829596/real 1787829596] req@ffff9e6804420f00 x1874668790742656/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829596.354335:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790742656/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829596.354339:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829596.354341:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829596.354343:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790742656:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829596.354366:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790742784:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829596.354376:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829596.354381:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829596/real 1787829596] req@ffff9e6804422580 x1874668790742784/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829596.354388:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790742784/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829596.354393:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829596.354394:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829596.354396:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790742784:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829596.354399:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790742912:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829596.354407:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829596.354412:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829596/real 1787829596] req@ffff9e6804421680 x1874668790742912/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829596.354424:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790742912/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829596.354428:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829596.354430:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829596.354431:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790742912:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829596.354434:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790743040:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829596.354442:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829596.354447:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829596/real 1787829596] req@ffff9e6804422940 x1874668790743040/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829596.354454:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790743040/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829596.354460:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829596.354461:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829596.354462:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790743040:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829596.354465:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790743168:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829596.354474:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829596.354479:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829596/real 1787829596] req@ffff9e6804420780 x1874668790743168/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829596.354484:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790743168/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829596.354499:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829596.354501:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829596.354503:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790743168:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829596.354506:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790743296:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829596.354536:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829596.354543:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829596/real 1787829596] req@ffff9e68044221c0 x1874668790743296/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829596.354550:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790743296/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829596.354555:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829596.354557:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829596.354559:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790743296:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829596.354562:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790743424:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829596.354571:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829596.354579:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829596/real 1787829596] req@ffff9e6804423840 x1874668790743424/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829596.354589:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790743424/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829596.354597:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829596.354600:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829596.354603:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790743424:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829596.865840:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829597.057900:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829597.889836:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829598.081907:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829598.913771:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829599.105899:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829599.937787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829600.129893:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829600.961800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829601.153869:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829601.473833:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829601.473843:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829601.473847:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829601.473857:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6238 00000100:00080000:1.0:1787829601.473869:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829601.473880:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829601.473923:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790743552/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829601.473947:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829601.473951:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829601.473952:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829601.473955:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6238 00000100:00080000:1.0:1787829601.473958:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829601.473961:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829601.473971:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829601.473980:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790743552:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787829601.473982:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790743680/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829601.473996:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829601.473998:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829601.473999:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829601.474001:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6238 00000100:00080000:1.0:1787829601.474003:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829601.474005:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829601.474008:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829601.474016:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790743808/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829601.474021:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829601.474024:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829601.474025:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829601.474027:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6238 00000100:00080000:1.0:1787829601.474029:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829601.474031:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829601.474034:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829601.474040:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790743936/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829601.474050:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829601.474053:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829601.474054:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829601.474056:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6238 00000100:00080000:1.0:1787829601.474059:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829601.474060:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829601.474063:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829601.474065:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829601.474074:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790744064/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829601.474081:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829601.474084:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829601.474086:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829601.474088:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6238 00000100:00080000:1.0:1787829601.474090:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829601.474093:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829601.474095:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829601.474105:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790744192/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829601.474110:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829601/real 1787829601] req@ffff9e6831aa8000 x1874668790743552/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829601.474113:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829601.474118:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829601.474121:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829601.474127:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6238 00000100:00080000:1.0:1787829601.474130:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829601.474133:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829601.474136:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790743552/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829601.474137:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829601.474147:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790744320/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829601.474148:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829601.474156:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829601.474157:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829601.474161:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790743552:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829601.474162:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829601.474165:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829601.474170:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6238 00000100:00080000:1.0:1787829601.474172:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829601.474175:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829601.474178:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829601.474184:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790744448/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829601.474188:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790743680:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829601.474197:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829601.474200:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829601.474201:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829601.474205:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6238 00000100:00080000:1.0:1787829601.474207:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829601.474210:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787829601.474213:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1787829601.474218:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829601.474224:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829601/real 1787829601] req@ffff9e6831aa9a40 x1874668790743680/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829601.474229:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790744576/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829601.474242:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790743680/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829601.474250:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829601.474252:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829601.474256:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790743680:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829601.474260:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790743808:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829601.474271:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829601.474283:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829601/real 1787829601] req@ffff9e6831aabc00 x1874668790743808/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829601.474291:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790743808/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829601.474299:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829601.474302:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829601.474305:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790743808:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829601.474311:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790743936:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829601.474332:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829601.474343:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829601/real 1787829601] req@ffff9e6831aa9680 x1874668790743936/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829601.474355:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790743936/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829601.474363:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829601.474366:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829601.474369:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790743936:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829601.474373:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790744064:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829601.474386:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829601.474393:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829601/real 1787829601] req@ffff9e6831aaa940 x1874668790744064/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829601.474399:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790744064/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829601.474409:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829601.474411:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829601.474413:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790744064:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829601.474415:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790744192:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829601.474425:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829601.474430:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829601/real 1787829601] req@ffff9e6831aa8780 x1874668790744192/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829601.474437:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790744192/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829601.474443:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829601.474444:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829601.474446:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790744192:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829601.474449:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790744320:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829601.474458:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829601.474463:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829601/real 1787829601] req@ffff9e6831aaa1c0 x1874668790744320/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829601.474469:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790744320/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829601.474485:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829601.474487:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829601.474490:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790744320:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829601.474503:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790744448:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829601.474517:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829601.474523:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829601/real 1787829601] req@ffff9e6831aaad00 x1874668790744448/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829601.474537:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790744448/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829601.474542:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829601.474544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829601.474546:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790744448:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829601.474549:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790744576:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829601.474559:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829601.474564:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829601/real 1787829601] req@ffff9e6831aab480 x1874668790744576/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829601.474571:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790744576/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829601.474575:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829601.474577:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829601.474578:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790744576:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829601.985786:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829602.177898:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829603.009812:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829603.201830:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829604.033818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829604.225916:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829605.057829:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829605.249931:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829606.081819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829606.273842:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829606.593816:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829606.593827:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829606.593831:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829606.593842:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6243 00000100:00080000:1.0:1787829606.593854:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829606.593862:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829606.593915:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790744704/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829606.593945:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829606.593948:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829606.593949:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829606.593952:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6243 00000100:00080000:1.0:1787829606.593954:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829606.593957:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829606.593966:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829606.593976:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790744832/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829606.593986:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829606.593989:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829606.593990:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829606.593993:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6243 00000100:00080000:1.0:1787829606.593995:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829606.593998:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829606.594001:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829606.594008:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790744960/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829606.594016:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829606.594020:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829606.594021:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829606.594023:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6243 00000100:00080000:1.0:1787829606.594026:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829606.594031:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790744704:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829606.594032:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829606.594039:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829606.594047:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790745088/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829606.594053:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829606.594056:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829606.594058:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829606.594060:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6243 00000100:00080000:1.0:1787829606.594062:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829606.594064:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829606.594067:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829606.594082:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790745216/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829606.594092:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829606.594096:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829606.594099:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829606.594103:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6243 00000100:00080000:1.0:1787829606.594108:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829606.594113:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829606.594117:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829606.594124:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829606.594138:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790745344/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829606.594150:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829606.594152:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829606.594154:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829606.594158:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6243 00000100:00100000:2.0:1787829606.594163:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829606/real 1787829606] req@ffff9e6831aaa580 x1874668790744704/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829606.594164:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829606.594171:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829606.594178:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829606.594188:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790745472/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829606.594192:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790744704/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829606.594207:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829606.594212:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829606.594214:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829606.594215:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829606.594219:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6243 00000100:00080000:1.0:1787829606.594222:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829606.594223:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829606.594228:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829606.594232:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790744704:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787829606.594234:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829606.594245:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790744832:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829606.594248:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790745600/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829606.594257:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829606.594260:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829606.594262:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829606.594265:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6243 00000100:00080000:1.0:1787829606.594269:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829606.594270:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829606.594274:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829606.594279:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829606.594285:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829606/real 1787829606] req@ffff9e6831aa8f00 x1874668790744832/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829606.594291:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790745728/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829606.594307:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790744832/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829606.594315:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829606.594318:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829606.594322:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790744832:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829606.594327:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790744960:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829606.594340:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829606.594346:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829606/real 1787829606] req@ffff9e6831aa9e00 x1874668790744960/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829606.594358:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790744960/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829606.594364:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829606.594366:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829606.594367:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790744960:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829606.594390:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790745088:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829606.594401:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829606.594407:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829606/real 1787829606] req@ffff9e6831aa8b40 x1874668790745088/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829606.594414:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790745088/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829606.594420:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829606.594422:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829606.594423:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790745088:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829606.594427:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790745216:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829606.594437:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829606.594460:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829606/real 1787829606] req@ffff9e6831aab480 x1874668790745216/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829606.594468:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790745216/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829606.594478:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829606.594480:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829606.594482:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790745216:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829606.594486:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790745344:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829606.594500:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829606.594506:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829606/real 1787829606] req@ffff9e6831aaad00 x1874668790745344/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829606.594514:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790745344/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829606.594520:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829606.594522:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829606.594524:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790745344:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829606.594528:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790745472:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829606.594539:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829606.594544:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829606/real 1787829606] req@ffff9e6831aaa1c0 x1874668790745472/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829606.594552:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790745472/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829606.594556:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829606.594558:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829606.594561:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790745472:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829606.594565:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790745600:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829606.594587:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829606.594594:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829606/real 1787829606] req@ffff9e6831aa8780 x1874668790745600/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829606.594602:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790745600/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829606.594607:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829606.594609:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829606.594611:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790745600:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829606.594615:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790745728:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829606.594626:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829606.594631:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829606/real 1787829606] req@ffff9e6831aaa940 x1874668790745728/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829606.594639:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790745728/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829606.594644:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829606.594645:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829606.594647:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790745728:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829607.105789:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829607.297922:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829608.129840:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829608.321862:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829609.153789:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829609.345914:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829610.177845:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829610.369892:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829611.201813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829611.393873:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829611.713815:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829611.713827:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829611.713832:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829611.713843:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6248 00000100:00080000:1.0:1787829611.713854:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829611.713861:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829611.713911:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790745856/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829611.713942:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829611.713945:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829611.713946:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829611.713950:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6248 00000100:00080000:1.0:1787829611.713952:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829611.713955:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829611.713964:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829611.713975:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790745984/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829611.713984:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829611.713988:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829611.713990:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829611.713992:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790745856:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829611.713995:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6248 00000100:00080000:1.0:1787829611.714003:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829611.714005:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829611.714008:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829611.714015:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790746112/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829611.714022:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829611.714025:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829611.714026:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829611.714029:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6248 00000100:00080000:1.0:1787829611.714037:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829611.714040:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829611.714044:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829611.714053:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790746240/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829611.714060:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829611.714063:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829611.714065:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829611.714067:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6248 00000100:00080000:1.0:1787829611.714070:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829611.714073:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829611.714076:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829611.714082:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829611.714086:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790746368/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829611.714094:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829611.714097:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829611.714098:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829611.714101:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6248 00000100:00080000:1.0:1787829611.714104:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829611.714107:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829611.714109:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829611.714116:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790746496/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829611.714121:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829611.714124:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829611.714126:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829611.714129:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6248 00000100:00080000:1.0:1787829611.714132:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829611.714135:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829611.714138:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829611.714145:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829611/real 1787829611] req@ffff9e6831aa9680 x1874668790745856/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829611.714147:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790746624/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829611.714180:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829611.714182:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829611.714183:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829611.714186:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6248 00000100:00080000:1.0:1787829611.714188:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829611.714193:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790745856/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829611.714194:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829611.714200:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829611.714210:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790746752/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829611.714217:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829611.714218:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829611.714226:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829611.714229:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829611.714230:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829611.714240:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6248 00000100:00100000:2.0:1787829611.714240:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790745856:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829611.714248:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829611.714255:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829611.714260:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829611.714274:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790745984:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829611.714289:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790746880/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829611.714302:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829611.714316:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829611/real 1787829611] req@ffff9e6831aabc00 x1874668790745984/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829611.714326:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790745984/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829611.714332:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829611.714335:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829611.714338:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790745984:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829611.714342:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790746112:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829611.714354:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829611.714360:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829611/real 1787829611] req@ffff9e6831aa9a40 x1874668790746112/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829611.714381:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790746112/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829611.714390:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829611.714393:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829611.714397:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790746112:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829611.714417:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790746240:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829611.714442:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829611.714455:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829611/real 1787829611] req@ffff9e6831aa8000 x1874668790746240/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829611.714469:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790746240/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829611.714478:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829611.714482:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829611.714485:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790746240:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829611.714492:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790746368:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829611.714511:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829611.714520:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829611/real 1787829611] req@ffff9e6831aaa940 x1874668790746368/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829611.714532:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790746368/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829611.714548:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829611.714550:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829611.714553:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790746368:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829611.714557:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790746496:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829611.714571:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829611.714579:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829611/real 1787829611] req@ffff9e6831aa8780 x1874668790746496/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829611.714586:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790746496/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829611.714593:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829611.714595:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829611.714597:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790746496:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829611.714600:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790746624:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829611.714611:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829611.714617:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829611/real 1787829611] req@ffff9e6831aaa1c0 x1874668790746624/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829611.714624:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790746624/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829611.714629:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829611.714631:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829611.714633:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790746624:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829611.714637:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790746752:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829611.714647:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829611.714653:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829611/real 1787829611] req@ffff9e6831aaad00 x1874668790746752/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829611.714660:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790746752/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829611.714665:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829611.714666:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829611.714668:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790746752:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829611.714684:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790746880:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829611.714717:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829611.714730:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829611/real 1787829611] req@ffff9e6831aab480 x1874668790746880/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829611.714742:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790746880/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829611.714751:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829611.714754:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829611.714758:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790746880:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829612.225844:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829612.417926:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829613.249790:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829613.441912:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829614.273769:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829614.465871:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829615.297861:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829615.489894:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829616.321778:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829616.513839:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829616.833897:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829616.833907:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829616.833911:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829616.833919:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6253 00000100:00080000:1.0:1787829616.833930:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829616.833936:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829616.833970:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790747008/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829616.834000:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829616.834002:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829616.834004:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829616.834006:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6253 00000100:00080000:1.0:1787829616.834009:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829616.834011:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829616.834019:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829616.834029:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790747136/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829616.834039:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829616.834041:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829616.834042:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829616.834047:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6253 00000100:00100000:2.0:1787829616.834049:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790747008:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829616.834050:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829616.834057:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829616.834062:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829616.834074:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790747264/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829616.834085:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829616.834089:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829616.834092:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829616.834096:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6253 00000100:00080000:1.0:1787829616.834102:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829616.834106:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829616.834112:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829616.834119:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829616.834127:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790747392/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829616.834137:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829616.834139:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829616.834141:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829616.834144:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6253 00000100:00080000:1.0:1787829616.834147:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829616.834150:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829616.834154:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829616.834163:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829616/real 1787829616] req@ffff9e6831aa8b40 x1874668790747008/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829616.834166:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790747520/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829616.834194:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829616.834197:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829616.834199:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829616.834203:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790747008/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829616.834205:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6253 00000100:00080000:1.0:1787829616.834212:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829616.834218:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787829616.834221:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787829616.834222:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829616.834227:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829616.834233:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790747648/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829616.834234:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790747008:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829616.834247:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829616.834250:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829616.834251:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829616.834255:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6253 00000100:00080000:1.0:1787829616.834258:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829616.834262:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790747136:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829616.834263:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829616.834270:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829616.834281:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790747776/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829616.834290:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829616.834293:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829616.834296:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829616.834300:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829616.834307:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6253 00000100:00100000:2.0:1787829616.834309:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829616/real 1787829616] req@ffff9e6831aa9e00 x1874668790747136/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829616.834315:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829616.834324:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790747136/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829616.834327:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829616.834339:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829616.834340:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829616.834346:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829616.834350:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790747136:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829616.834356:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790747264:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829616.834360:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790747904/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829616.834390:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829616.834393:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1787829616.834398:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829616.834399:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829616.834407:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6253 00000100:00080000:1.0:1787829616.834412:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829616.834413:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829616/real 1787829616] req@ffff9e6831aa8f00 x1874668790747264/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829616.834423:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829616.834429:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790747264/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829616.834431:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829616.834443:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829616.834446:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829616.834453:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790747264:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829616.834454:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790748032/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829616.834459:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790747392:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829616.834481:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829616.834491:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829616/real 1787829616] req@ffff9e6831aaa580 x1874668790747392/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829616.834499:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790747392/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829616.834505:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829616.834506:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829616.834508:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790747392:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829616.834512:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790747520:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829616.834522:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829616.834528:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829616/real 1787829616] req@ffff9e6831aab480 x1874668790747520/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829616.834540:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790747520/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829616.834545:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829616.834547:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829616.834549:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790747520:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829616.834552:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790747648:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829616.834561:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829616.834567:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829616/real 1787829616] req@ffff9e6831aaad00 x1874668790747648/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829616.834573:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790747648/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829616.834579:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829616.834582:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829616.834583:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790747648:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829616.834596:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790747776:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829616.834606:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829616.834612:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829616/real 1787829616] req@ffff9e6831aaa1c0 x1874668790747776/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829616.834619:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790747776/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829616.834628:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829616.834630:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829616.834632:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790747776:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829616.834636:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790747904:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829616.834644:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829616.834650:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829616/real 1787829616] req@ffff9e6831aa8780 x1874668790747904/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829616.834657:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790747904/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829616.834662:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829616.834664:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829616.834665:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790747904:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829616.834669:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790748032:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829616.834679:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829616.834688:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829616/real 1787829616] req@ffff9e6831aaa940 x1874668790748032/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829616.834729:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790748032/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829616.834738:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829616.834742:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829616.834746:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790748032:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829617.345793:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829617.537823:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829618.369796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829618.561838:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829619.393769:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829619.585858:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829620.417794:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829620.609854:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829621.441769:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829621.633909:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829621.953795:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829621.953803:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829621.953808:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829621.953817:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6258 00000100:00080000:3.0:1787829621.953826:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829621.953832:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829621.953865:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790748160/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829621.953885:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829621.953888:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829621.953889:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829621.953892:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6258 00000100:00080000:3.0:1787829621.953894:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829621.953897:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829621.953904:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829621.953911:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790748288/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829621.953922:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829621.953924:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829621.953926:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829621.953928:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6258 00000100:00080000:3.0:1787829621.953930:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829621.953932:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829621.953936:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829621.953942:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790748416/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829621.953948:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829621.953950:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829621.953952:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829621.953954:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6258 00000100:00080000:3.0:1787829621.953956:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829621.953962:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790748160:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829621.953962:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829621.953967:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829621.953977:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790748544/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829621.953983:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829621.953985:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829621.953986:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829621.953989:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6258 00000100:00080000:3.0:1787829621.953991:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829621.953994:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829621.953997:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829621.954005:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790748672/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829621.954011:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829621.954014:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829621.954015:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829621.954018:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6258 00000100:00080000:3.0:1787829621.954020:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829621.954023:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829621.954026:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829621.954029:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829621.954038:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790748800/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829621.954045:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829621.954047:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829621.954049:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829621.954051:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6258 00000100:00080000:3.0:1787829621.954053:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829621.954056:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829621.954059:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829621.954068:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829621/real 1787829621] req@ffff9e680a2d5a40 x1874668790748160/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829621.954070:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790748928/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829621.954099:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829621.954102:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790748160/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829621.954103:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829621.954107:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829621.954116:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6258 00000100:00080000:3.0:1787829621.954120:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829621.954121:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829621.954124:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787829621.954129:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:3.0:1787829621.954130:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829621.954134:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790748160:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829621.954141:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790749056/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829621.954144:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790748288:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829621.954162:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829621.954167:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829621.954168:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829621.954171:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829621.954176:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6258 00000100:00080000:3.0:1787829621.954179:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829621.954181:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829621/real 1787829621] req@ffff9e680a2d7480 x1874668790748288/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829621.954185:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829621.954192:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829621.954198:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790748288/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829621.954203:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790749184/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829621.954208:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829621.954217:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829621.954222:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790748288:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829621.954228:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790748416:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829621.954243:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829621.954250:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829621/real 1787829621] req@ffff9e680a2d4b40 x1874668790748416/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829621.954257:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790748416/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829621.954262:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829621.954264:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829621.954266:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790748416:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829621.954286:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790748544:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829621.954297:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829621.954303:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829621/real 1787829621] req@ffff9e680a2d4f00 x1874668790748544/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829621.954311:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790748544/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829621.954316:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829621.954318:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829621.954320:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790748544:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829621.954323:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790748672:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829621.954349:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829621.954359:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829621/real 1787829621] req@ffff9e680a2d7c00 x1874668790748672/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829621.954367:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790748672/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829621.954372:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829621.954374:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829621.954376:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790748672:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829621.954380:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790748800:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829621.954392:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829621.954398:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829621/real 1787829621] req@ffff9e680a2d5e00 x1874668790748800/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829621.954413:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790748800/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829621.954422:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829621.954424:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829621.954426:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790748800:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829621.954430:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790748928:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829621.954441:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829621.954447:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829621/real 1787829621] req@ffff9e680a2d6d00 x1874668790748928/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829621.954454:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790748928/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829621.954460:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829621.954462:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829621.954464:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790748928:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829621.954467:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790749056:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829621.954477:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829621.954482:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829621/real 1787829621] req@ffff9e680a2d61c0 x1874668790749056/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829621.954489:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790749056/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829621.954494:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829621.954496:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829621.954498:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790749056:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829621.954501:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790749184:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829621.954511:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829621.954516:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829621/real 1787829621] req@ffff9e680a2d52c0 x1874668790749184/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829621.954524:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790749184/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829646 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829621.954528:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829621.954530:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829621.954532:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790749184:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829622.465840:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829622.657933:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829623.489794:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829623.681924:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829624.513834:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829624.705876:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829625.537854:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829625.729855:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829626.561840:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829626.753940:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829627.073831:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829627.073851:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829627.073860:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829627.073879:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6263 00000100:00080000:3.0:1787829627.073898:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829627.073912:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829627.073969:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790749312/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829627.074023:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829627.074029:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829627.074033:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829627.074041:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6263 00000100:00080000:3.0:1787829627.074046:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829627.074054:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829627.074069:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829627.074094:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790749440/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829627.074110:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829627.074116:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829627.074119:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829627.074124:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6263 00000100:00080000:3.0:1787829627.074131:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829627.074140:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790749312:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829627.074142:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829627.074160:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829627.074182:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790749568/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829627.074211:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829627.074216:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829627.074220:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829627.074227:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6263 00000800:00000100:1.0:1787829627.074237:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829627.074290:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829627/real 1787829627] req@ffff9e680a2d4780 x1874668790749312/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829627.074337:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829627.074348:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829627.074352:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790749312/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829627.074359:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829627.074392:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829627.074403:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829627.074412:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790749312:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787829627.074430:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790749440:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829627.074461:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829627.074478:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829627/real 1787829627] req@ffff9e680a2d43c0 x1874668790749440/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829627.074527:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790749440/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829627.074537:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829627.074540:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829627.074542:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790749696/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829627.074544:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790749440:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829627.074572:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829627.074579:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829627.074587:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790749568:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829627.074611:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829627.074632:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829627/real 1787829627] req@ffff9e680a2d70c0 x1874668790749568/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829627.074639:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829627.074654:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6263 00000100:00100000:1.0:1787829627.074656:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790749568/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829627.074666:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829627.074675:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829627.074685:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787829627.074691:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829627.074698:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790749568:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829627.074743:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790749696:192.168.203.109@tcp:8:kworker.0 00000008:00100000:3.0:1787829627.074745:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829627.074780:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829627.074781:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790749824/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829627.074798:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829627/real 1787829627] req@ffff9e680a2d6580 x1874668790749696/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829627.074806:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829627.074813:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790749696/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829627.074819:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829627.074829:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829627.074847:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6263 00000100:00080000:3.0:1787829627.074865:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829627.074868:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829627.074878:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787829627.074879:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829627.074897:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790749696:192.168.203.109@tcp:8:kworker.0 00000008:00100000:3.0:1787829627.074898:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829627.074925:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790749952/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829627.074938:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790749824:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829627.074956:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829627.074965:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:1.0:1787829627.074970:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829627.074974:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829627.074984:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829627/real 1787829627] req@ffff9e680a2d4000 x1874668790749824/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829627.074987:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6263 00000100:00080000:3.0:1787829627.075005:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829627.075015:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790749824/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829627.075023:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829627.075038:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829627.075071:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829627.075081:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790750080/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829627.075086:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829627.075108:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790749824:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829627.075136:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829627.075143:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787829627.075147:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790749952:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829627.075151:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829627.075163:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6263 00000100:00080000:3.0:1787829627.075171:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829627.075181:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829627.075182:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829627.075192:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829627.075200:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829627/real 1787829627] req@ffff9e680a2d43c0 x1874668790749952/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829627.075212:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790750208/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829627.075219:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790749952/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829627.075315:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829627.075329:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829627.075330:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829627.075344:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829627.075347:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829627.075352:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790749952:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829627.075356:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6263 00000100:00080000:3.0:1787829627.075365:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829627.075373:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829627.075381:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790750080:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787829627.075381:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829627.075401:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790750336/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829627.075435:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829627.075451:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829627/real 1787829627] req@ffff9e680a2d4780 x1874668790750080/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829627.075463:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790750080/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829627.075471:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829627.075474:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829627.075477:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790750080:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829627.075492:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790750208:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829627.075509:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829627.075518:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829627/real 1787829627] req@ffff9e680a2d52c0 x1874668790750208/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829627.075529:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790750208/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829627.075536:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829627.075538:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829627.075541:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790750208:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829627.075546:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790750336:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829627.075560:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829627.075569:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829627/real 1787829627] req@ffff9e680a2d61c0 x1874668790750336/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829627.075579:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790750336/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829627.075586:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829627.075589:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829627.075592:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790750336:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829627.585841:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829627.777907:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829628.609806:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829628.801849:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829629.633798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829629.825901:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829630.657795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829630.849865:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829631.681809:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829631.873854:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829632.193773:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829632.193782:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829632.193786:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829632.193795:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6268 00000100:00080000:3.0:1787829632.193805:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829632.193811:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829632.193845:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790750464/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829632.193872:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829632.193876:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829632.193879:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829632.193884:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6268 00000100:00080000:3.0:1787829632.193888:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829632.193893:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829632.193904:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829632.193921:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790750592/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829632.193922:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790750464:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829632.193940:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829632.193944:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829632.193946:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829632.193950:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6268 00000100:00080000:3.0:1787829632.193953:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829632.193957:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829632.193961:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829632.193970:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790750720/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829632.193985:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829632.193987:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829632.193989:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829632.193992:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6268 00000100:00080000:3.0:1787829632.193994:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829632.193997:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829632.194001:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829632.194012:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790750848/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829632.194018:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829632.194021:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829632.194022:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829632.194028:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6268 00000100:00100000:1.0:1787829632.194029:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790750592:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829632.194031:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829632.194036:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829632.194042:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829632.194052:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790750976/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829632.194060:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829632.194063:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829632.194064:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829632.194067:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6268 00000100:00100000:1.0:1787829632.194071:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790750720:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829632.194073:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829632.194080:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829632.194084:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829632.194095:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790751104/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829632.194102:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829632.194104:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829632.194105:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829632.194110:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790750848:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829632.194110:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6268 00000100:00080000:3.0:1787829632.194113:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829632.194120:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829632.194125:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829632.194134:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790751232/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829632.194142:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829632.194144:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790750976:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829632.194145:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829632.194149:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829632.194154:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6268 00000100:00080000:3.0:1787829632.194157:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829632.194160:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829632.194164:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829632.194170:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790751104:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829632.194175:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790751360/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829632.194183:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829632.194186:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829632.194188:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829632.194192:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6268 00000100:00100000:1.0:1787829632.194195:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790751232:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829632.194196:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829632.194202:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829632.194206:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829632.194215:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790751488/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829632.194219:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790751360:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829632.194259:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790751488:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829632.705797:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829632.897893:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829633.729841:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829633.921879:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829634.753800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829634.945845:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787829635.266137:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:1.0:1787829635.266189:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787829635.266208:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829635.266299:0:2387:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:1.0:1787829635.266321:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787829635.266368:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829632/real 1787829635] req@ffff9e680a2d52c0 x1874668790751488/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787829635.266379:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787829635.266429:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790751488/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787829635.266438:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829635.266467:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00080000:2.0:1787829635.266476:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829635.266488:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000800:00000100:1.0:1787829635.266499:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787829635.266503:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790751488:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829635.266522:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787829635.266533:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829632/real 1787829635] req@ffff9e680a2d4b40 x1874668790750976/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787829635.266556:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787829635.266566:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790750976/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829635.266589:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829635.266598:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829635.266603:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790750976:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829635.266612:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829632/real 1787829635] req@ffff9e680a2d7480 x1874668790751104/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829635.266628:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790751104/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829635.266639:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829635.266643:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829635.266646:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790751104:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829635.266655:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829632/real 1787829635] req@ffff9e680a2d5e00 x1874668790750592/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829635.266667:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790750592/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829635.266676:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829635.266680:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829635.266683:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790750592:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829635.266737:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829632/real 1787829635] req@ffff9e680a2d7c00 x1874668790750720/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829635.266758:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790750720/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829635.266770:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829635.266775:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829635.266780:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790750720:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829635.266795:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829632/real 1787829635] req@ffff9e680a2d4f00 x1874668790750848/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829635.266808:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790750848/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829635.266816:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829635.266820:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829635.266822:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790750848:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829635.266827:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829632/real 1787829635] req@ffff9e680a2d6d00 x1874668790750464/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829635.266837:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790750464/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829635.266846:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829635.266850:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829635.266853:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790750464:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829635.266903:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829632/real 1787829635] req@ffff9e680a2d5a40 x1874668790751232/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829635.266918:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790751232/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829635.266927:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829635.266932:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829635.266936:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790751232:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829635.266944:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829632/real 1787829635] req@ffff9e680a2d61c0 x1874668790751360/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829635.266958:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790751360/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829635.266967:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829635.266971:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829635.266974:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790751360:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829635.777905:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829635.969857:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829636.801825:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829636.993849:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829637.313769:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829637.313783:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829637.313788:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829637.313801:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6274 00000100:00080000:3.0:1787829637.313815:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829637.313824:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829637.313873:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790751616/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829637.313904:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829637.313907:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829637.313908:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829637.313912:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6274 00000100:00080000:3.0:1787829637.313915:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829637.313919:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829637.313929:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829637.313951:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790751744/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829637.313954:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790751616:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829637.313963:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829637.313967:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829637.313970:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829637.313975:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6274 00000100:00080000:3.0:1787829637.313978:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829637.313983:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829637.313988:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829637.314001:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790751872/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829637.314011:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829637.314015:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829637.314017:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829637.314021:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6274 00000100:00080000:3.0:1787829637.314026:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829637.314030:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829637.314036:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829637.314047:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829637.314050:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790752000/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829637.314062:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829637.314066:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829637.314068:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829637.314073:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6274 00000100:00080000:3.0:1787829637.314077:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829637.314081:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829637.314087:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829637.314102:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790752128/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829637.314109:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829637/real 1787829637] req@ffff9e680a2d4780 x1874668790751616/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829637.314115:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829637.314125:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829637.314129:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829637.314134:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6274 00000100:00080000:3.0:1787829637.314139:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829637.314141:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790751616/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829637.314147:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829637.314158:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829637.314163:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829637.314170:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829637.314176:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790752256/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829637.314181:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790751616:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829637.314199:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829637.314203:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829637.314205:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829637.314210:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6274 00000100:00080000:3.0:1787829637.314215:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829637.314219:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829637.314240:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829637.314242:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790751744:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829637.314255:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790752384/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829637.314267:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829637.314271:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1787829637.314276:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829637.314276:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829637.314286:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6274 00000100:00080000:3.0:1787829637.314291:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829637.314296:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829637.314302:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829637/real 1787829637] req@ffff9e680a2d43c0 x1874668790751744/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829637.314304:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829637.314323:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790752512/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829637.314325:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790751744/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829637.314352:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829637.314352:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829637.314362:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829637.314362:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829637.314367:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829637.314371:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790751744:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829637.314377:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6274 00000100:00100000:2.0:1787829637.314383:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790751872:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829637.314384:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829637.314394:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829637.314400:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829637.314411:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829637.314419:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790752640/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829637.314426:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829637/real 1787829637] req@ffff9e680a2d4000 x1874668790751872/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829637.314445:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790751872/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829637.314453:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829637.314458:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829637.314462:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790751872:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829637.314468:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790752000:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829637.314488:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829637.314498:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829637/real 1787829637] req@ffff9e680a2d6580 x1874668790752000/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829637.314511:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790752000/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829637.314520:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829637.314523:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829637.314527:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790752000:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829637.314533:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790752128:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829637.314550:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829637.314559:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829637/real 1787829637] req@ffff9e680a2d70c0 x1874668790752128/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829637.314571:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790752128/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829637.314579:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829637.314583:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829637.314586:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790752128:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829637.314592:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790752256:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829637.314610:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829637.314621:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829637/real 1787829637] req@ffff9e680a2d61c0 x1874668790752256/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829637.314631:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790752256/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829637.314648:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829637.314651:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829637.314654:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790752256:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829637.314676:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790752384:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829637.314727:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829637.314740:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829637/real 1787829637] req@ffff9e680a2d5a40 x1874668790752384/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829637.314753:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790752384/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829637.314762:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829637.314765:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829637.314769:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790752384:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829637.314775:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790752512:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829637.314793:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829637.314803:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829637/real 1787829637] req@ffff9e680a2d6d00 x1874668790752512/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829637.314815:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790752512/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829637.314823:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829637.314826:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829637.314830:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790752512:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829637.314837:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790752640:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829637.314854:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829637.314870:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829637/real 1787829637] req@ffff9e680a2d4f00 x1874668790752640/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829637.314882:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790752640/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829637.314890:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829637.314894:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829637.314897:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790752640:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829637.825794:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829638.017875:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829638.849802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829639.041901:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829639.873822:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829640.065898:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829640.897774:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829641.089939:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829641.921833:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829642.113894:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829642.433786:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829642.433796:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829642.433801:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829642.433811:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6279 00000100:00080000:3.0:1787829642.433824:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829642.433834:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829642.433887:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790752768/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829642.433916:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829642.433919:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829642.433920:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829642.433924:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6279 00000100:00080000:3.0:1787829642.433927:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829642.433930:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829642.433940:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829642.433950:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790752896/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829642.433959:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829642.433961:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790752768:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829642.433963:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829642.433966:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829642.433972:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6279 00000100:00080000:3.0:1787829642.433974:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829642.433976:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829642.433979:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829642.433988:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790753024/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829642.433994:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829642.433997:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829642.433998:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829642.434002:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6279 00000100:00080000:3.0:1787829642.434005:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829642.434008:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829642.434012:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829642.434019:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790753152/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829642.434025:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829642.434028:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829642.434029:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829642.434032:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6279 00000100:00080000:3.0:1787829642.434034:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829642.434036:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829642.434038:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829642.434048:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829642.434048:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790753280/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829642.434057:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829642.434060:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829642.434061:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829642.434064:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6279 00000100:00080000:3.0:1787829642.434067:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829642.434070:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829642.434073:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829642.434080:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790753408/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829642.434087:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829642.434090:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829642.434091:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829642.434095:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6279 00000100:00080000:3.0:1787829642.434096:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829642.434099:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829642.434104:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829642/real 1787829642] req@ffff9e680a2d7c00 x1874668790752768/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829642.434104:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829642.434119:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790753536/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829642.434126:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790752768/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829642.434128:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829642.434137:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829642.434140:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829642.434145:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6279 00000100:00080000:2.0:1787829642.434146:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829642.434148:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829642.434152:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829642.434154:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829642.434160:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829642.434161:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790752768:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829642.434171:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790753664/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829642.434178:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829642.434180:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829642.434181:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829642.434184:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6279 00000100:00080000:3.0:1787829642.434187:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829642.434203:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790752896:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829642.434209:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829642.434214:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829642.434226:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790753792/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829642.434235:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829642.434248:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829642/real 1787829642] req@ffff9e680a2d5e00 x1874668790752896/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829642.434257:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790752896/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829642.434264:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829642.434266:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829642.434268:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790752896:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829642.434273:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790753024:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829642.434284:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829642.434290:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829642/real 1787829642] req@ffff9e680a2d7480 x1874668790753024/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829642.434298:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790753024/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829642.434303:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829642.434305:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829642.434307:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790753024:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829642.434311:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790753152:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829642.434321:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829642.434328:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829642/real 1787829642] req@ffff9e680a2d4b40 x1874668790753152/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829642.434335:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790753152/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829642.434341:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829642.434343:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829642.434345:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790753152:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829642.434348:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790753280:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829642.434357:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829642.434363:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829642/real 1787829642] req@ffff9e680a2d52c0 x1874668790753280/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829642.434369:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790753280/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829642.434374:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829642.434377:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829642.434379:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790753280:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829642.434382:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790753408:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829642.434392:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829642.434397:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829642/real 1787829642] req@ffff9e680a2d4f00 x1874668790753408/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829642.434404:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790753408/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829642.434411:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829642.434413:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829642.434415:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790753408:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829642.434418:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790753536:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829642.434428:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829642.434433:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829642/real 1787829642] req@ffff9e680a2d6d00 x1874668790753536/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829642.434440:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790753536/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829642.434445:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829642.434447:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829642.434449:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790753536:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829642.434452:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790753664:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829642.434462:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829642.434468:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829642/real 1787829642] req@ffff9e680a2d5a40 x1874668790753664/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829642.434474:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790753664/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829642.434480:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829642.434482:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829642.434484:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790753664:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829642.434501:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790753792:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829642.434513:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829642.434519:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829642/real 1787829642] req@ffff9e680a2d61c0 x1874668790753792/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829642.434526:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790753792/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829642.434531:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829642.434533:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829642.434535:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790753792:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829642.945760:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829643.137855:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829643.969795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829644.161861:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829644.993846:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829645.185909:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829646.017804:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829646.209963:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829647.041817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829647.233853:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829647.553790:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829647.553800:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829647.553806:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829647.553816:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6284 00000100:00080000:3.0:1787829647.553830:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829647.553840:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829647.553889:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790753920/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829647.553917:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829647.553920:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829647.553921:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829647.553925:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6284 00000100:00080000:3.0:1787829647.553928:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829647.553931:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829647.553940:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829647.553949:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790754048/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829647.553957:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829647.553959:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829647.553961:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829647.553963:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6284 00000100:00080000:3.0:1787829647.553965:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829647.553967:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829647.553970:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829647.553978:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790754176/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829647.553984:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829647.553986:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829647.553987:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829647.553990:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6284 00000100:00080000:3.0:1787829647.553992:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829647.553994:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829647.553999:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829647.554001:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790753920:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829647.554008:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790754304/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829647.554017:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829647.554019:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829647.554020:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829647.554022:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6284 00000100:00080000:3.0:1787829647.554026:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829647.554029:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829647.554032:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829647.554041:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790754432/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829647.554047:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829647.554050:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829647.554052:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829647.554054:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6284 00000100:00080000:3.0:1787829647.554065:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829647.554069:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829647.554072:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829647.554082:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829647.554084:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790754560/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829647.554092:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829647.554095:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829647.554096:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829647.554099:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6284 00000100:00080000:3.0:1787829647.554102:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829647.554105:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829647.554108:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829647.554115:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790754688/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829647.554127:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829647.554131:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787829647.554132:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829647/real 1787829647] req@ffff9e680a2d70c0 x1874668790753920/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829647.554133:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829647.554142:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6284 00000100:00080000:3.0:1787829647.554147:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829647.554151:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829647.554168:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829647.554176:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790753920/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829647.554183:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790754816/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829647.554190:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829647.554193:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829647.554197:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829647.554198:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829647.554202:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829647.554205:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790753920:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829647.554209:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6284 00000100:00080000:3.0:1787829647.554214:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829647.554216:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790754048:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829647.554220:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829647.554225:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829647.554234:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790754944/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829647.554245:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829647.554258:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829647/real 1787829647] req@ffff9e680a2d6580 x1874668790754048/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829647.554266:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790754048/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829647.554273:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829647.554275:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829647.554277:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790754048:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829647.554281:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790754176:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829647.554291:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829647.554297:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829647/real 1787829647] req@ffff9e680a2d4000 x1874668790754176/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829647.554304:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790754176/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829647.554309:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829647.554311:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829647.554313:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790754176:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829647.554336:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790754304:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829647.554347:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829647.554354:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829647/real 1787829647] req@ffff9e680a2d43c0 x1874668790754304/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829647.554361:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790754304/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829647.554367:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829647.554368:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829647.554370:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790754304:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829647.554373:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790754432:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829647.554383:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829647.554389:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829647/real 1787829647] req@ffff9e680a2d4780 x1874668790754432/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829647.554396:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790754432/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829647.554401:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829647.554403:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829647.554405:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790754432:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829647.554408:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790754560:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829647.554417:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829647.554423:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829647/real 1787829647] req@ffff9e680a2d61c0 x1874668790754560/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829647.554430:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790754560/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829647.554438:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829647.554440:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829647.554442:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790754560:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829647.554445:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790754688:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829647.554455:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829647.554460:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829647/real 1787829647] req@ffff9e680a2d5a40 x1874668790754688/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829647.554467:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790754688/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829647.554473:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829647.554474:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829647.554476:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790754688:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829647.554479:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790754816:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829647.554497:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829647.554504:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829647/real 1787829647] req@ffff9e680a2d6d00 x1874668790754816/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829647.554511:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790754816/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829647.554516:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829647.554518:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829647.554521:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790754816:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829647.554525:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790754944:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829647.554541:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829647.554547:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829647/real 1787829647] req@ffff9e680a2d4f00 x1874668790754944/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829647.554554:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790754944/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829647.554560:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829647.554561:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829647.554563:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790754944:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829648.065800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829648.257888:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829649.089821:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829649.281830:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829650.113785:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829650.305863:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829651.137825:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829651.329851:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829652.161820:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829652.353828:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829652.673822:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829652.673837:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829652.673842:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829652.673856:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6289 00000100:00080000:1.0:1787829652.673869:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829652.673877:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829652.673923:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790755072/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829652.673950:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829652.673953:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829652.673954:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829652.673957:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6289 00000100:00080000:1.0:1787829652.673960:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829652.673964:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829652.673980:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829652.673990:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790755072:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787829652.673995:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790755200/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829652.674004:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829652.674007:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829652.674008:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829652.674011:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6289 00000100:00080000:1.0:1787829652.674014:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829652.674016:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829652.674020:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829652.674029:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790755328/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829652.674036:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829652.674039:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829652.674040:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829652.674043:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6289 00000100:00080000:1.0:1787829652.674045:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829652.674048:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829652.674051:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829652.674058:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790755456/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829652.674064:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829652.674068:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829652.674070:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829652.674073:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829652.674077:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6289 00000100:00080000:1.0:1787829652.674081:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829652.674084:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829652.674087:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829652.674096:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790755584/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829652.674102:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829652.674106:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829652.674107:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829652.674110:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6289 00000100:00080000:1.0:1787829652.674113:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829652.674130:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829652.674132:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829652/real 1787829652] req@ffff9e6831aa8000 x1874668790755072/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829652.674135:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829652.674150:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790755712/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829652.674157:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790755072/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829652.674159:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829652.674168:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829652.674172:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829652.674177:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6289 00000100:00080000:2.0:1787829652.674179:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829652.674181:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829652.674185:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829652.674189:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829652.674194:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790755072:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787829652.674195:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829652.674207:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790755840/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829652.674213:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829652.674215:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829652.674217:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829652.674222:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790755200:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829652.674223:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6289 00000100:00080000:1.0:1787829652.674229:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829652.674233:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829652.674236:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829652.674246:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790755968/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829652.674253:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829652.674254:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829652.674259:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829652.674261:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829652.674265:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6289 00000100:00080000:1.0:1787829652.674269:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829652.674270:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829652/real 1787829652] req@ffff9e6831aa9a40 x1874668790755200/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829652.674274:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829652.674282:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829652.674288:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790755200/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829652.674293:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790756096/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829652.674303:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829652.674309:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829652.674313:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790755200:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829652.674317:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790755328:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829652.674331:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829652.674338:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829652/real 1787829652] req@ffff9e6831aabc00 x1874668790755328/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829652.674345:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790755328/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829652.674350:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829652.674352:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829652.674354:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790755328:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829652.674357:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790755456:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829652.674368:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829652.674374:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829652/real 1787829652] req@ffff9e6831aa9680 x1874668790755456/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829652.674381:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790755456/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829652.674387:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829652.674388:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829652.674390:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790755456:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829652.674393:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790755584:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829652.674403:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829652.674408:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829652/real 1787829652] req@ffff9e6831aaa940 x1874668790755584/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829652.674415:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790755584/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829652.674421:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829652.674422:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829652.674424:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790755584:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829652.674427:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790755712:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829652.674437:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829652.674443:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829652/real 1787829652] req@ffff9e6831aa8780 x1874668790755712/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829652.674449:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790755712/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829652.674456:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829652.674458:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829652.674460:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790755712:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829652.674463:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790755840:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829652.674482:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829652.674488:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829652/real 1787829652] req@ffff9e6831aaa1c0 x1874668790755840/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829652.674495:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790755840/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829652.674500:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829652.674501:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829652.674503:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790755840:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829652.674516:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790755968:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829652.674527:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829652.674533:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829652/real 1787829652] req@ffff9e6831aaad00 x1874668790755968/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829652.674541:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790755968/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829652.674545:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829652.674547:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829652.674550:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790755968:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829652.674554:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790756096:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829652.674564:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829652.674569:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829652/real 1787829652] req@ffff9e6831aab480 x1874668790756096/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829652.674576:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790756096/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829652.674581:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829652.674583:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829652.674585:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790756096:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829653.185814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829653.377944:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829654.209802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829654.401876:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829655.233826:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:0.0:1787829655.426034:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829656.257812:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829656.449880:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829657.281814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829657.473852:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829657.793845:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829657.793858:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829657.793864:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829657.793877:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6294 00000100:00080000:0.0:1787829657.793891:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829657.793899:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829657.793951:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790756224/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829657.793984:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829657.793988:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829657.793990:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829657.793996:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6294 00000100:00080000:0.0:1787829657.794000:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829657.794005:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829657.794018:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829657.794034:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790756352/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829657.794046:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790756224:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829657.794049:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829657.794058:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829657.794060:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829657.794066:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6294 00000100:00080000:0.0:1787829657.794070:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829657.794074:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829657.794080:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829657.794112:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790756480/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829657.794141:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829657.794146:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829657.794148:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829657.794154:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6294 00000100:00080000:0.0:1787829657.794160:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829657.794165:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829657.794172:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829657.794174:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829657.794192:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790756608/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829657.794204:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829657.794208:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829657.794211:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829657.794216:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6294 00000100:00080000:0.0:1787829657.794221:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829657.794226:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829657.794230:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829657/real 1787829657] req@ffff9e6803e9ad00 x1874668790756224/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829657.794235:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829657.794258:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790756736/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829657.794259:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790756224/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829657.794275:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829657.794282:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829657.794284:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829657.794285:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829657.794292:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829657.794296:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6294 00000100:00100000:2.0:1787829657.794301:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790756224:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829657.794304:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829657.794314:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829657.794320:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829657.794334:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790756864/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829657.794336:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790756352:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829657.794378:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829657.794384:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829657.794386:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829657.794391:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829657.794397:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6294 00000100:00100000:2.0:1787829657.794401:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829657/real 1787829657] req@ffff9e6803e98f00 x1874668790756352/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829657.794404:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829657.794430:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829657.794435:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790756352/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829657.794441:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829657.794446:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829657.794451:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829657.794454:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790756352:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829657.794463:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790756480:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829657.794465:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790756992/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829657.794479:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829657.794483:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1787829657.794492:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829657.794494:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829657.794501:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6294 00000100:00080000:0.0:1787829657.794506:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829657.794524:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829657/real 1787829657] req@ffff9e6803e992c0 x1874668790756480/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829657.794531:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829657.794541:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829657.794556:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790756480/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829657.794558:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790757120/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829657.794608:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829657.794611:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829657.794612:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829657.794617:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829657.794619:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829657.794623:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6294 00000100:00100000:2.0:1787829657.794626:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790756480:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829657.794628:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829657.794636:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829657.794640:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790756608:192.168.203.109@tcp:8:kworker.0 00000008:00100000:0.0:1787829657.794643:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829657.794657:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790757248/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829657.794669:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829657.794682:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829657/real 1787829657] req@ffff9e6803e9bc00 x1874668790756608/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829657.794692:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790756608/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829657.794699:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829657.794701:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829657.794703:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790756608:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829657.794707:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790756736:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829657.794751:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829657.794767:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829657/real 1787829657] req@ffff9e6803e9a940 x1874668790756736/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829657.794781:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790756736/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829657.794788:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829657.794790:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829657.794792:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790756736:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829657.794807:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790756864:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829657.794823:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829657.794831:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829657/real 1787829657] req@ffff9e6803e98b40 x1874668790756864/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829657.794839:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790756864/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829657.794847:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829657.794850:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829657.794852:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790756864:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829657.794856:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790756992:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829657.794867:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829657.794872:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829657/real 1787829657] req@ffff9e6803e98780 x1874668790756992/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829657.794880:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790756992/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829657.794886:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829657.794888:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829657.794889:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790756992:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829657.794893:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790757120:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829657.794903:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829657.794909:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829657/real 1787829657] req@ffff9e6803e9a1c0 x1874668790757120/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829657.794916:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790757120/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829657.794921:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829657.794923:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829657.794926:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790757120:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829657.794931:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790757248:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829657.794941:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829657.794947:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829657/real 1787829657] req@ffff9e6803e9b840 x1874668790757248/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829657.794955:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790757248/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829657.794960:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829657.794962:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829657.794964:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790757248:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829658.305779:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829658.497856:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829659.329798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829659.521851:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829660.353798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829660.545850:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829661.377810:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829661.569890:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829662.401774:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829662.593867:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829662.913924:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829662.913934:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829662.913939:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829662.913949:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6299 00000100:00080000:2.0:1787829662.913963:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829662.913973:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829662.914015:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790757376/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829662.914043:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829662.914060:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829662.914064:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829662.914071:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6299 00000100:00080000:2.0:1787829662.914079:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829662.914087:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829662.914101:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829662.914105:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790757376:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829662.914127:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790757504/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829662.914141:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829662.914144:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829662.914147:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829662.914151:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6299 00000100:00080000:2.0:1787829662.914156:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829662.914165:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829662.914170:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829662.914177:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829662.914183:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790757632/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829662.914197:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829662.914202:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829662.914204:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829662.914211:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6299 00000100:00080000:2.0:1787829662.914215:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829662.914221:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829662.914230:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829662.914236:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829662/real 1787829662] req@ffff9e6804421e00 x1874668790757376/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829662.914245:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790757760/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829662.914261:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829662.914265:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829662.914269:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829662.914278:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790757376/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829662.914279:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6299 00000100:00080000:2.0:1787829662.914291:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829662.914304:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787829662.914307:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787829662.914310:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829662.914317:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829662.914325:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790757376:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829662.914328:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790757888/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829662.914359:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829662.914362:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790757504:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829662.914364:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829662.914370:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829662.914380:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6299 00000100:00080000:2.0:1787829662.914384:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829662.914387:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829662.914392:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829662.914403:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790758016/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829662.914412:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829662.914421:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829662.914422:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829662.914428:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829662.914435:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6299 00000100:00080000:2.0:1787829662.914443:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829662.914448:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829662.914452:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829662.914457:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829662/real 1787829662] req@ffff9e68044212c0 x1874668790757504/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829662.914467:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790758144/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829662.914476:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790757504/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829662.914479:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829662.914499:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829662.914505:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829662.914516:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6299 00000100:00080000:1.0:1787829662.914520:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829662.914520:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829662.914524:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829662.914527:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829662.914536:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790758272/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829662.914545:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829662.914556:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829662.914563:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790757504:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829662.914565:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829662.914571:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829662.914577:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790757632:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829662.914581:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6299 00000100:00080000:2.0:1787829662.914591:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829662.914597:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829662.914607:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829662.914615:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829662.914627:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790758400/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829662.914637:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829662/real 1787829662] req@ffff9e68044230c0 x1874668790757632/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829662.914679:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790757632/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829662.914686:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829662.914690:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829662.914692:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790757632:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829662.914722:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790757760:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829662.914746:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829662.914756:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829662/real 1787829662] req@ffff9e6804420b40 x1874668790757760/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829662.914764:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790757760/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829662.914772:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829662.914775:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829662.914779:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790757760:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829662.914797:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790757888:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829662.914819:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829662.914829:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829662/real 1787829662] req@ffff9e6804423840 x1874668790757888/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829662.914837:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790757888/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829662.914848:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829662.914850:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829662.914852:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790757888:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829662.914857:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790758016:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829662.914870:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829662.914876:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829662/real 1787829662] req@ffff9e68044221c0 x1874668790758016/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829662.914883:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790758016/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829662.914889:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829662.914891:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829662.914893:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790758016:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829662.914897:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790758144:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829662.914907:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829662.914913:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829662/real 1787829662] req@ffff9e6804420780 x1874668790758144/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829662.914920:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790758144/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829662.914925:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829662.914927:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829662.914929:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790758144:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829662.914933:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790758272:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829662.914943:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829662.914949:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829662/real 1787829662] req@ffff9e6804422940 x1874668790758272/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829662.914956:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790758272/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829662.914961:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829662.914963:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829662.914965:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790758272:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829662.914968:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790758400:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829662.914978:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829662.914984:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829662/real 1787829662] req@ffff9e6804421680 x1874668790758400/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829662.914990:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790758400/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829662.914995:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829662.914997:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829662.914999:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790758400:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829663.425807:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829663.617957:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829664.449819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829664.641896:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829665.473815:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829665.665956:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829666.497870:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829666.689901:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829667.521847:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829667.713909:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829668.033827:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829668.033841:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829668.033846:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829668.033875:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6304 00000100:00080000:3.0:1787829668.033890:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829668.033898:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829668.033944:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790758528/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829668.033988:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829668.033991:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829668.033996:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829668.034000:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6304 00000100:00080000:3.0:1787829668.034002:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829668.034006:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829668.034028:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829668.034042:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790758656/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829668.034048:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790758528:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829668.034074:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829668.034077:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829668.034078:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829668.034082:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6304 00000100:00080000:3.0:1787829668.034085:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829668.034088:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829668.034091:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829668.034100:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790758784/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829668.034107:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829668.034110:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829668.034111:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829668.034114:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6304 00000100:00080000:3.0:1787829668.034117:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829668.034119:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829668.034122:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829668.034130:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790758912/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829668.034136:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829668.034138:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829668.034141:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829668.034143:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829668.034147:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6304 00000100:00080000:3.0:1787829668.034151:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829668.034155:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829668.034158:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829668.034167:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790759040/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829668.034173:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829668.034175:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829668.034177:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829668.034181:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6304 00000100:00080000:3.0:1787829668.034183:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829668.034187:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829668.034190:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829668.034192:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829668/real 1787829668] req@ffff9e680a2d52c0 x1874668790758528/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829668.034202:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790759168/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829668.034210:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829668.034213:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790758528/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829668.034214:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829668.034217:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829668.034225:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6304 00000100:00080000:3.0:1787829668.034229:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829668.034233:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829668.034233:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829668.034239:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829668.034254:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829668.034258:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790759296/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829668.034260:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790758528:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829668.034278:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829668.034280:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829668.034282:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829668.034285:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6304 00000100:00080000:3.0:1787829668.034287:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829668.034293:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829668.034294:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790758656:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787829668.034296:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829668.034318:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790759424/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829668.034326:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829668.034331:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829668.034343:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829668/real 1787829668] req@ffff9e680a2d4b40 x1874668790758656/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829668.034346:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829668.034351:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829668.034358:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790758656/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829668.034362:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6304 00000100:00080000:3.0:1787829668.034372:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829668.034374:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829668.034381:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787829668.034382:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829668.034388:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790758656:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787829668.034393:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829668.034396:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790758784:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829668.034414:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790759552/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829668.034418:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829668.034428:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829668/real 1787829668] req@ffff9e680a2d7480 x1874668790758784/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829668.034443:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790758784/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829668.034450:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829668.034452:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829668.034454:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790758784:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829668.034458:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790758912:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829668.034471:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829668.034477:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829668/real 1787829668] req@ffff9e680a2d5e00 x1874668790758912/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829668.034498:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790758912/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829668.034519:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829668.034523:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829668.034526:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790758912:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829668.034533:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790759040:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829668.034553:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829668.034561:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829668/real 1787829668] req@ffff9e680a2d7c00 x1874668790759040/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829668.034569:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790759040/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829668.034580:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829668.034582:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829668.034584:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790759040:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829668.034587:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790759168:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829668.034599:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829668.034605:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829668/real 1787829668] req@ffff9e680a2d4f00 x1874668790759168/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829668.034612:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790759168/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829668.034619:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829668.034620:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829668.034622:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790759168:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829668.034626:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790759296:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829668.034637:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829668.034642:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829668/real 1787829668] req@ffff9e680a2d6d00 x1874668790759296/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829668.034650:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790759296/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829668.034655:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829668.034656:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829668.034659:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790759296:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829668.034676:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790759424:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829668.034687:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829668.034693:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829668/real 1787829668] req@ffff9e680a2d5a40 x1874668790759424/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829668.034724:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790759424/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829668.034733:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829668.034737:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829668.034740:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790759424:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829668.034747:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790759552:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829668.034770:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829668.034782:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829668/real 1787829668] req@ffff9e680a2d61c0 x1874668790759552/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829668.034792:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790759552/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829668.034798:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829668.034800:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829668.034802:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790759552:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829668.545839:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829668.737885:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829669.569785:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829669.761891:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829670.593816:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829670.785889:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829671.617802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829671.809851:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829672.641820:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829672.833859:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829673.153804:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829673.153818:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829673.153824:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829673.153838:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6309 00000100:00080000:0.0:1787829673.153850:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829673.153858:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829673.153905:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790759680/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829673.153935:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829673.153940:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829673.153943:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829673.153949:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6309 00000100:00080000:0.0:1787829673.153953:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829673.153967:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829673.153970:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790759680:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1787829673.153993:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829673.154023:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790759808/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829673.154033:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829673.154038:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829673.154040:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829673.154042:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6309 00000100:00080000:0.0:1787829673.154053:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829673.154057:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829673.154061:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829673.154065:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829673.154074:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790759936/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829673.154083:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829673.154086:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829673.154088:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829673.154090:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6309 00000100:00080000:0.0:1787829673.154093:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829673.154096:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829673.154099:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829673.154111:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790760064/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829673.154117:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829673.154122:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829673.154123:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829673/real 1787829673] req@ffff9e6803e9a580 x1874668790759680/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829673.154124:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829673.154130:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6309 00000100:00080000:0.0:1787829673.154135:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829673.154140:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829673.154145:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829673.154151:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790759680/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829673.154156:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790760192/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829673.154171:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829673.154175:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829673.154178:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829673.154183:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787829673.154184:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790759680:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829673.154189:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829673.154197:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6309 00000100:00080000:0.0:1787829673.154202:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829673.154208:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829673.154213:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790759808:192.168.203.109@tcp:38:kworker.0 00000008:00100000:0.0:1787829673.154214:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829673.154234:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790760320/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829673.154242:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829673.154251:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829673.154255:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829673.154268:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829673.154274:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829673/real 1787829673] req@ffff9e6803e99680 x1874668790759808/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829673.154275:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6309 00000100:00080000:0.0:1787829673.154282:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829673.154290:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829673.154298:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829673.154302:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790759808/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829673.154312:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829673.154314:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790760448/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829673.154318:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829673.154325:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790759808:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829673.154342:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829673.154345:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790759936:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829673.154346:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829673.154351:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829673.154355:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6309 00000100:00080000:0.0:1787829673.154359:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829673.154362:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829673.154367:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829673.154372:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829673.154379:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790760576/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829673.154386:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829673.154389:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829673/real 1787829673] req@ffff9e6803e99e00 x1874668790759936/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829673.154404:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829673.154407:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829673.154412:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6309 00000100:00080000:0.0:1787829673.154417:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829673.154422:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829673.154430:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829673.154439:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790759936/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829673.154445:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790760704/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829673.154450:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829673.154460:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829673.154468:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790759936:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829673.154477:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790760064:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829673.154502:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829673.154512:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829673/real 1787829673] req@ffff9e6803e9b480 x1874668790760064/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829673.154520:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790760064/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829673.154526:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829673.154529:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829673.154531:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790760064:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829673.154543:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790760192:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829673.154559:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829673.154576:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829673/real 1787829673] req@ffff9e6803e9b840 x1874668790760192/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829673.154584:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790760192/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829673.154594:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829673.154596:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829673.154598:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790760192:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829673.154611:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790760320:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829673.154621:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829673.154628:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829673/real 1787829673] req@ffff9e6803e9a1c0 x1874668790760320/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829673.154634:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790760320/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829673.154641:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829673.154643:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829673.154645:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790760320:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829673.154649:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790760448:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829673.154669:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829673.154674:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829673/real 1787829673] req@ffff9e6803e98780 x1874668790760448/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829673.154682:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790760448/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829673.154687:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829673.154688:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829673.154691:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790760448:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829673.154713:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790760576:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829673.154735:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829673.154746:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829673/real 1787829673] req@ffff9e6803e98b40 x1874668790760576/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829673.154758:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790760576/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829673.154766:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829673.154770:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829673.154774:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790760576:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829673.154780:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790760704:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829673.154798:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829673.154806:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829673/real 1787829673] req@ffff9e6803e9a940 x1874668790760704/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829673.154831:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790760704/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829673.154837:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829673.154839:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829673.154841:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790760704:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829673.665808:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829673.857855:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829674.689823:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829674.881912:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829675.713780:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829675.905883:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829676.737830:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829676.929908:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829677.761814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829677.953894:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829678.273817:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829678.273828:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829678.273833:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829678.273844:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6314 00000100:00080000:3.0:1787829678.273856:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829678.273863:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829678.273908:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790760832/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829678.273935:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829678.273937:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829678.273952:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829678.273957:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6314 00000100:00080000:3.0:1787829678.273962:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829678.273966:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829678.273977:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829678.273980:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790760832:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829678.273991:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790760960/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829678.274000:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829678.274002:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829678.274004:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829678.274007:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6314 00000100:00080000:3.0:1787829678.274009:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829678.274013:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829678.274016:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829678.274025:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790761088/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829678.274031:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829678.274034:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829678.274035:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829678.274038:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6314 00000100:00080000:3.0:1787829678.274040:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829678.274042:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829678.274045:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829678.274052:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790761216/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829678.274058:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829678.274061:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829678.274062:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829678.274068:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6314 00000800:00000100:1.0:1787829678.274070:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829678.274071:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829678.274076:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829678.274079:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829678.274087:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790761344/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829678.274094:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829678.274097:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829678.274099:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829678.274101:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6314 00000100:00080000:3.0:1787829678.274104:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829678.274106:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829678.274109:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829678.274118:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790761472/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829678.274124:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829678.274127:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829678.274130:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829678.274132:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829678/real 1787829678] req@ffff9e680a2d4780 x1874668790760832/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829678.274134:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6314 00000100:00080000:3.0:1787829678.274147:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829678.274154:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829678.274157:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829678.274165:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790760832/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829678.274169:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790761600/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829678.274188:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829678.274189:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829678.274191:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829678.274193:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829678.274196:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6314 00000100:00080000:3.0:1787829678.274201:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829678.274208:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829678.274208:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829678.274215:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829678.274218:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790760832:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829678.274226:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790761728/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829678.274233:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829678.274235:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829678.274237:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829678.274240:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6314 00000100:00080000:3.0:1787829678.274243:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829678.274249:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790760960:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829678.274250:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829678.274256:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829678.274267:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790761856/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829678.274283:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829678.274302:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829678/real 1787829678] req@ffff9e680a2d43c0 x1874668790760960/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829678.274311:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790760960/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829678.274318:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829678.274320:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829678.274322:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790760960:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829678.274327:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790761088:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829678.274340:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829678.274346:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829678/real 1787829678] req@ffff9e680a2d4000 x1874668790761088/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829678.274359:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790761088/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829678.274364:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829678.274366:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829678.274368:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790761088:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829678.274372:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790761216:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829678.274383:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829678.274388:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829678/real 1787829678] req@ffff9e680a2d6580 x1874668790761216/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829678.274395:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790761216/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829678.274401:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829678.274403:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829678.274404:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790761216:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829678.274408:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790761344:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829678.274418:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829678.274423:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829678/real 1787829678] req@ffff9e680a2d70c0 x1874668790761344/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829678.274431:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790761344/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829678.274443:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829678.274447:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829678.274450:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790761344:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829678.274456:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790761472:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829678.274477:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829678.274488:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829678/real 1787829678] req@ffff9e680a2d61c0 x1874668790761472/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829678.274499:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790761472/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829678.274507:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829678.274509:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829678.274511:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790761472:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829678.274515:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790761600:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829678.274530:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829678.274537:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829678/real 1787829678] req@ffff9e680a2d5a40 x1874668790761600/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829678.274544:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790761600/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829678.274549:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829678.274551:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829678.274554:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790761600:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829678.274558:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790761728:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829678.274568:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829678.274574:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829678/real 1787829678] req@ffff9e680a2d6d00 x1874668790761728/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829678.274581:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790761728/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829678.274586:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829678.274588:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829678.274590:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790761728:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829678.274606:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790761856:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829678.274617:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829678.274623:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829678/real 1787829678] req@ffff9e680a2d4f00 x1874668790761856/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829678.274630:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790761856/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829678.274635:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829678.274637:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829678.274638:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790761856:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829678.785834:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829678.977929:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829679.809795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829680.001896:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829680.833819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829681.025891:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829681.857798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829682.049841:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829682.881788:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829683.073936:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829683.393835:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829683.393845:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829683.393850:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829683.393861:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6320 00000100:00080000:0.0:1787829683.393874:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829683.393881:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829683.393935:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790761984/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829683.393971:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829683.393975:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829683.393978:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829683.393983:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6320 00000100:00080000:0.0:1787829683.393988:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829683.393993:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829683.394003:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829683.394003:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790761984:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787829683.394025:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790762112/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829683.394036:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829683.394040:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829683.394043:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829683.394047:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6320 00000100:00080000:0.0:1787829683.394052:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829683.394057:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1787829683.394062:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787829683.394065:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829683.394080:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790762240/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829683.394090:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829683.394094:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829683.394097:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829683.394098:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829683/real 1787829683] req@ffff9e6803e9bc00 x1874668790761984/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829683.394105:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6320 00000100:00080000:0.0:1787829683.394113:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829683.394121:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790761984/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829683.394122:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829683.394132:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829683.394135:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829683.394140:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829683.394144:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790761984:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787829683.394149:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790762368/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829683.394161:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829683.394164:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829683.394166:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790762112:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829683.394168:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829683.394175:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6320 00000100:00080000:0.0:1787829683.394180:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829683.394185:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1787829683.394189:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787829683.394194:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829683.394201:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829683/real 1787829683] req@ffff9e6803e992c0 x1874668790762112/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829683.394211:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790762496/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829683.394214:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790762112/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829683.394241:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829683.394242:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829683.394245:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829683.394252:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787829683.394252:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790762112:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829683.394257:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829683.394261:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790762240:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829683.394267:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6320 00000100:00080000:0.0:1787829683.394273:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829683.394281:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1787829683.394281:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787829683.394290:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829683.394292:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829683/real 1787829683] req@ffff9e6803e98f00 x1874668790762240/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829683.394306:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790762624/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829683.394312:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790762240/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829683.394333:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829683.394341:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829683.394345:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829683.394346:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790762240:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829683.394355:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787829683.394358:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790762368:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787829683.394359:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829683.394367:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6320 00000100:00080000:0.0:1787829683.394374:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829683.394377:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829683.394382:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829683.394387:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829683/real 1787829683] req@ffff9e6803e9ad00 x1874668790762368/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829683.394389:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829683.394402:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790762368/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829683.394409:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790762752/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829683.394410:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829683.394416:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829683.394423:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790762368:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787829683.394432:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829683.394436:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829683.394439:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829683.394440:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790762496:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787829683.394447:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6320 00000100:00080000:0.0:1787829683.394455:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829683.394458:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829683.394464:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829683.394469:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829683/real 1787829683] req@ffff9e6803e9bc00 x1874668790762496/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829683.394472:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829683.394483:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790762496/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829683.394491:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790762880/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829683.394498:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829683.394506:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829683.394512:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790762496:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787829683.394516:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829683.394519:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790762624:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829683.394522:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829683.394525:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829683.394531:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6320 00000800:00000100:1.0:1787829683.394535:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829683.394544:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829683/real 1787829683] req@ffff9e6803e9a940 x1874668790762624/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829683.394546:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829683.394554:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829683.394561:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829683.394561:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790762624/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829683.394572:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829683.394576:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790763008/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829683.394578:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829683.394589:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790762624:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829683.394598:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790762752:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829683.394619:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829683.394632:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829683/real 1787829683] req@ffff9e6803e98b40 x1874668790762752/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829683.394644:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790762752/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829683.394649:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829683.394651:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829683.394654:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790762752:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829683.394665:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790762880:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829683.394679:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829683.394686:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829683/real 1787829683] req@ffff9e6803e98780 x1874668790762880/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829683.394693:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790762880/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829683.394719:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829683.394721:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829683.394724:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790762880:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829683.394728:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790763008:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829683.394740:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829683.394746:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829683/real 1787829683] req@ffff9e6803e9a1c0 x1874668790763008/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829683.394754:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790763008/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829683.394759:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829683.394761:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829683.394763:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790763008:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829683.905862:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829684.097968:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829684.929797:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829685.121860:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829685.953817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829686.145847:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829686.977783:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829687.169947:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829688.001864:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829688.193854:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829688.513772:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829688.513781:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829688.513786:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829688.513795:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6325 00000100:00080000:0.0:1787829688.513804:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829688.513811:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829688.513844:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790763136/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829688.513866:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829688.513884:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829688.513885:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829688.513889:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6325 00000100:00080000:0.0:1787829688.513893:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829688.513897:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829688.513906:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829688.513913:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790763136:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787829688.513921:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790763264/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829688.513931:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829688.513936:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829688.513938:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829688.513941:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6325 00000100:00080000:0.0:1787829688.513944:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829688.513947:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829688.513950:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829688.513959:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790763392/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829688.513966:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829688.513969:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829688.513971:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829688.513976:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829688.513980:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6325 00000100:00080000:0.0:1787829688.513984:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829688.513987:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829688.513991:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829688.514000:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790763520/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829688.514008:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829688.514012:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829688.514016:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829688/real 1787829688] req@ffff9e6803e9b840 x1874668790763136/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829688.514017:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829688.514029:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6325 00000100:00080000:0.0:1787829688.514037:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829688.514041:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790763136/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829688.514046:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787829688.514059:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787829688.514060:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829688.514065:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829688.514070:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790763136:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787829688.514080:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790763648/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829688.514092:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829688.514094:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790763264:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829688.514096:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829688.514102:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829688.514123:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6325 00000100:00080000:0.0:1787829688.514126:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829688.514130:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829688.514135:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829688.514138:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829688.514158:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829688/real 1787829688] req@ffff9e6803e9b480 x1874668790763264/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829688.514161:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790763776/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829688.514183:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829688.514185:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790763264/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829688.514188:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829688.514192:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829688.514201:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6325 00000100:00080000:1.0:1787829688.514206:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829688.514207:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829688.514214:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787829688.514215:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829688.514219:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790763264:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829688.514225:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790763392:192.168.203.109@tcp:38:kworker.0 00000008:00100000:0.0:1787829688.514227:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829688.514245:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829688.514246:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790763904/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829688.514255:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829688/real 1787829688] req@ffff9e6803e99e00 x1874668790763392/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829688.514261:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829688.514266:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829688.514268:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829688.514279:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6325 00000100:00100000:1.0:1787829688.514284:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790763392/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829688.514297:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829688.514307:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829688.514308:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787829688.514313:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:0.0:1787829688.514318:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829688.514319:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790763392:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829688.514329:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790763520:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829688.514339:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790764032/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829688.514347:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829688.514359:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829688.514359:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829688/real 1787829688] req@ffff9e6803e99680 x1874668790763520/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829688.514370:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829688.514376:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829688.514378:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790763520/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829688.514396:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6325 00000100:00080000:1.0:1787829688.514404:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829688.514407:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829688.514409:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829688.514410:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790763520:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787829688.514417:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829688.514423:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829688.514428:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790763648:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829688.514437:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790764160/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829688.514463:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829688.514476:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829688/real 1787829688] req@ffff9e6803e9a580 x1874668790763648/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829688.514485:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790763648/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829688.514497:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829688.514499:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829688.514501:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790763648:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829688.514505:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790763776:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829688.514517:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829688.514523:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829688/real 1787829688] req@ffff9e6803e9b840 x1874668790763776/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829688.514531:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790763776/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829688.514538:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829688.514540:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829688.514542:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790763776:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829688.514545:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790763904:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829688.514556:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829688.514563:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829688/real 1787829688] req@ffff9e6803e9a1c0 x1874668790763904/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829688.514588:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790763904/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829688.514597:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829688.514600:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829688.514604:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790763904:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829688.514611:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790764032:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829688.514633:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829688.514661:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829688/real 1787829688] req@ffff9e6803e98780 x1874668790764032/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829688.514675:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790764032/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829688.514681:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829688.514683:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829688.514686:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790764032:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829688.514699:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790764160:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829688.514715:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829688.514723:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829688/real 1787829688] req@ffff9e6803e98b40 x1874668790764160/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829688.514731:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790764160/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829688.514737:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829688.514758:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829688.514761:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790764160:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829689.025760:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829689.217828:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829690.049834:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829690.241905:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829691.073789:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829691.265895:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829692.097861:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829692.289895:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829693.121792:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829693.313889:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829693.633801:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829693.633809:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829693.633813:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829693.633821:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6330 00000100:00080000:2.0:1787829693.633830:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829693.633849:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829693.633891:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790764288/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829693.633914:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829693.633916:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829693.633917:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829693.633920:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6330 00000100:00080000:2.0:1787829693.633923:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829693.633926:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829693.633948:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829693.633957:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790764288:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829693.633959:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790764416/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829693.633969:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829693.633972:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829693.633973:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829693.633976:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6330 00000100:00080000:2.0:1787829693.633978:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829693.633980:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829693.633984:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829693.633992:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790764544/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829693.633998:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829693.634001:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829693.634002:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829693.634004:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6330 00000100:00080000:2.0:1787829693.634006:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829693.634009:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829693.634012:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829693.634017:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829693.634024:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790764672/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829693.634031:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829693.634034:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829693.634035:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829693.634037:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6330 00000100:00080000:2.0:1787829693.634040:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829693.634042:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829693.634045:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829693.634052:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790764800/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829693.634058:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829693.634060:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829693.634062:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829693.634066:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6330 00000100:00100000:1.0:1787829693.634067:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829693/real 1787829693] req@ffff9e6804422580 x1874668790764288/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829693.634068:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829693.634073:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829693.634078:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829693.634090:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790764928/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829693.634095:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790764288/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829693.634099:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829693.634106:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829693.634109:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829693.634115:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829693.634115:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6330 00000100:00080000:2.0:1787829693.634120:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829693.634123:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829693.634127:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829693.634131:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790764288:192.168.203.109@tcp:250:kworker.0 00000008:00100000:2.0:1787829693.634131:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829693.634143:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790765056/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829693.634149:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829693.634151:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829693.634153:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829693.634156:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6330 00000100:00080000:2.0:1787829693.634158:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829693.634160:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829693.634163:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829693.634166:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790764416:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829693.634173:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790765184/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829693.634182:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829693.634184:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829693.634186:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829693.634190:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6330 00000800:00000100:1.0:1787829693.634193:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829693.634194:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829693.634199:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829693.634203:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829693.634209:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829693/real 1787829693] req@ffff9e6804420f00 x1874668790764416/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829693.634213:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790765312/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829693.634245:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790764416/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829693.634257:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829693.634261:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829693.634265:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790764416:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829693.634272:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790764544:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829693.634292:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829693.634300:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829693/real 1787829693] req@ffff9e6804423480 x1874668790764544/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829693.634310:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790764544/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829693.634315:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829693.634317:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829693.634320:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790764544:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829693.634324:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790764672:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829693.634335:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829693.634341:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829693/real 1787829693] req@ffff9e6804422d00 x1874668790764672/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829693.634349:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790764672/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829693.634354:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829693.634356:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829693.634358:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790764672:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829693.634361:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790764800:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829693.634371:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829693.634377:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829693/real 1787829693] req@ffff9e6804421680 x1874668790764800/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829693.634384:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790764800/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829693.634393:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829693.634395:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829693.634397:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790764800:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829693.634400:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790764928:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829693.634410:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829693.634415:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829693/real 1787829693] req@ffff9e6804422940 x1874668790764928/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829693.634422:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790764928/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829693.634428:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829693.634430:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829693.634432:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790764928:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829693.634435:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790765056:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829693.634444:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829693.634450:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829693/real 1787829693] req@ffff9e6804420780 x1874668790765056/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829693.634456:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790765056/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829693.634461:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829693.634462:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829693.634465:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790765056:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829693.634478:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790765184:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829693.634488:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829693.634494:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829693/real 1787829693] req@ffff9e68044221c0 x1874668790765184/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829693.634502:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790765184/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829693.634507:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829693.634509:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829693.634510:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790765184:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829693.634514:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790765312:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829693.634523:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829693.634528:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829693/real 1787829693] req@ffff9e6804423840 x1874668790765312/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829693.634536:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790765312/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829693.634540:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829693.634542:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829693.634544:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790765312:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829694.145842:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829694.337950:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829695.169798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829695.361853:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829696.193805:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829696.385927:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829697.217877:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829697.409937:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829698.241824:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829698.433912:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829698.753854:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829698.753866:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829698.753871:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829698.753883:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6335 00000100:00080000:3.0:1787829698.753895:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829698.753903:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829698.753949:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790765440/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829698.753990:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829698.753994:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829698.753995:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829698.754000:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6335 00000100:00080000:3.0:1787829698.754007:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829698.754012:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829698.754023:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829698.754031:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790765440:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829698.754039:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790765568/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829698.754048:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829698.754052:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829698.754053:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829698.754058:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6335 00000100:00080000:3.0:1787829698.754060:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829698.754063:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829698.754066:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829698.754075:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790765696/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829698.754081:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829698.754083:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829698.754085:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829698.754088:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6335 00000100:00080000:3.0:1787829698.754092:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829698.754095:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829698.754098:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829698.754112:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790765824/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829698.754127:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829698.754131:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829698.754134:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829698.754140:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6335 00000100:00080000:3.0:1787829698.754153:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829698.754159:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790765568:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829698.754162:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829698.754172:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829698.754187:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790765952/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829698.754214:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790765696:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829698.754243:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790765824:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829698.754273:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790765952:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829698.754298:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829698.754420:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829698.754423:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829698.754429:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6335 00000100:00080000:3.0:1787829698.754434:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829698.754439:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829698.754448:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829698.754468:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790766080/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829698.754493:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829698.754497:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829698.754500:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829698.754508:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6335 00000100:00080000:3.0:1787829698.754516:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829698.754521:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829698.754528:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829698.754541:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790766080:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829698.754551:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790766208/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829698.754567:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829698.754573:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829698.754576:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829698.754582:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6335 00000100:00080000:3.0:1787829698.754587:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829698.754594:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829698.754600:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790766208:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787829698.754603:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829698.754626:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790766336/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829698.754638:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829698.754644:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829698.754647:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829698.754652:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790766336:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829698.754655:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6335 00000100:00080000:3.0:1787829698.754663:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829698.754668:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829698.754674:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829698.754690:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790766464/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829698.754780:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790766464:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829699.265818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829699.458007:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829700.289851:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829700.481881:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829701.313877:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829701.505889:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787829701.826016:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:0.0:1787829701.826082:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829701.826104:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787829701.826167:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:0.0:1787829701.826182:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787829701.826192:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787829701.826203:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787829701.826218:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787829701.826232:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787829701.826246:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:0.0:1787829701.826264:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:1.0:1787829701.826283:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829698/real 1787829701] req@ffff9e680a2d61c0 x1874668790766464/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829701.826324:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790766464/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829701.826336:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829701.826341:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829701.826345:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790766464:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829701.826356:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829698/real 1787829701] req@ffff9e680a2d5a40 x1874668790766336/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829701.826367:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790766336/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829701.826375:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829701.826378:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829701.826381:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790766336:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829701.826389:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829698/real 1787829701] req@ffff9e680a2d6d00 x1874668790766208/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829701.826400:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790766208/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829701.826413:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829701.826417:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829701.826420:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790766208:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829701.826426:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829698/real 1787829701] req@ffff9e680a2d4f00 x1874668790766080/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829701.826438:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790766080/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829701.826448:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829701.826451:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829701.826455:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790766080:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829701.826463:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829698/real 1787829701] req@ffff9e680a2d52c0 x1874668790765952/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829701.826476:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790765952/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829701.826483:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829701.826486:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829701.826489:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790765952:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829701.826496:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829698/real 1787829701] req@ffff9e680a2d5e00 x1874668790765568/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829701.826508:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790765568/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829701.826517:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829701.826520:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829701.826523:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790765568:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829701.826531:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829698/real 1787829701] req@ffff9e680a2d7480 x1874668790765696/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829701.826543:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790765696/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829701.826551:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829701.826555:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829701.826559:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790765696:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829701.826566:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829698/real 1787829701] req@ffff9e680a2d4b40 x1874668790765824/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829701.826578:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790765824/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829701.826585:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829701.826589:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829701.826591:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790765824:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829701.826598:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829698/real 1787829701] req@ffff9e680a2d7c00 x1874668790765440/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829701.826611:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790765440/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829701.826620:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829701.826624:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829701.826627:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790765440:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829702.337838:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829702.529911:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829703.361871:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829703.553898:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829703.873828:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829703.873840:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829703.873845:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829703.873858:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6340 00000100:00080000:3.0:1787829703.873873:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829703.873883:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829703.873930:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790766592/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829703.873968:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829703.873972:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829703.873974:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829703.873978:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6340 00000100:00080000:3.0:1787829703.873982:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829703.873986:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829703.873999:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829703.874012:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790766720/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829703.874024:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829703.874030:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790766592:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829703.874031:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829703.874035:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829703.874045:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6340 00000100:00080000:3.0:1787829703.874049:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829703.874053:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829703.874057:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829703.874069:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790766848/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829703.874078:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829703.874081:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829703.874082:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829703.874085:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6340 00000100:00080000:3.0:1787829703.874088:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829703.874090:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829703.874094:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829703.874101:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790766976/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829703.874107:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829703.874110:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829703.874111:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829703.874113:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6340 00000100:00080000:3.0:1787829703.874116:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829703.874118:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829703.874122:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829703.874126:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829703.874134:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790767104/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829703.874145:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829703.874149:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829703.874151:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829703.874156:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6340 00000100:00080000:3.0:1787829703.874160:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829703.874165:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829703.874171:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829703.874186:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790767232/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829703.874190:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829703/real 1787829703] req@ffff9e680a2d70c0 x1874668790766592/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829703.874206:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829703.874212:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829703.874214:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829703.874220:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6340 00000100:00080000:3.0:1787829703.874225:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829703.874226:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790766592/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829703.874233:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829703.874243:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829703.874259:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829703.874264:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790767360/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829703.874269:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829703.874275:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790766592:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829703.874293:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829703.874298:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829703.874301:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829703.874307:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6340 00000100:00080000:3.0:1787829703.874312:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829703.874316:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790766720:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829703.874321:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829703.874330:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829703.874346:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790767488/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829703.874354:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829703.874365:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829703.874370:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829703.874372:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829703.874377:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829703/real 1787829703] req@ffff9e680a2d6580 x1874668790766720/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829703.874381:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6340 00000100:00080000:3.0:1787829703.874389:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829703.874397:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829703.874400:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790766720/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829703.874405:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829703.874422:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829703.874423:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790767616/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829703.874432:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829703.874440:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790766720:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829703.874451:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790766848:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829703.874476:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829703.874491:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829703/real 1787829703] req@ffff9e680a2d4000 x1874668790766848/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829703.874503:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790766848/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829703.874511:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829703.874515:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829703.874518:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790766848:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829703.874526:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790766976:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829703.874549:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829703.874561:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829703/real 1787829703] req@ffff9e680a2d43c0 x1874668790766976/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829703.874574:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790766976/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829703.874582:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829703.874586:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829703.874589:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790766976:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829703.874595:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790767104:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829703.874616:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829703.874628:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829703/real 1787829703] req@ffff9e680a2d4780 x1874668790767104/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829703.874641:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790767104/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829703.874650:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829703.874654:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829703.874657:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790767104:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829703.874665:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790767232:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829703.874686:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829703.874728:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829703/real 1787829703] req@ffff9e680a2d7c00 x1874668790767232/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829703.874743:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790767232/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829703.874753:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829703.874757:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829703.874760:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790767232:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829703.874783:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790767360:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829703.874810:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829703.874841:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829703/real 1787829703] req@ffff9e680a2d4b40 x1874668790767360/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829703.874854:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790767360/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829703.874863:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829703.874867:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829703.874871:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790767360:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829703.874906:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790767488:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829703.874932:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829703.874947:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829703/real 1787829703] req@ffff9e680a2d7480 x1874668790767488/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829703.874959:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790767488/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829703.874973:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829703.874976:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829703.874979:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790767488:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829703.874985:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790767616:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829703.875003:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829703.875015:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829703/real 1787829703] req@ffff9e680a2d5e00 x1874668790767616/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829703.875026:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790767616/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829703.875034:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829703.875037:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829703.875041:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790767616:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829704.385849:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829704.577973:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829705.409840:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829705.601871:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829706.433784:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829706.625847:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829707.457842:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829707.649959:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829708.481812:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829708.673870:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829708.993803:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829708.993815:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829708.993819:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829708.993831:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6345 00000100:00080000:3.0:1787829708.993843:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829708.993851:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829708.993895:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790767744/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829708.993925:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829708.993930:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829708.993932:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829708.993937:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6345 00000100:00080000:3.0:1787829708.993942:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829708.993948:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829708.993961:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829708.993974:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790767744:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829708.993976:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790767872/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829708.993994:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829708.993998:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829708.993999:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829708.994003:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6345 00000100:00080000:3.0:1787829708.994006:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829708.994010:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829708.994014:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829708.994023:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790768000/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829708.994030:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829708.994033:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829708.994035:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829708.994038:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6345 00000100:00080000:3.0:1787829708.994041:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829708.994044:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829708.994048:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829708.994050:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829708.994060:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790768128/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829708.994069:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829708.994071:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829708.994073:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829708.994076:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6345 00000100:00080000:3.0:1787829708.994078:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829708.994080:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829708.994083:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829708.994091:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790768256/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829708.994097:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829708.994101:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829708/real 1787829708] req@ffff9e680a2d52c0 x1874668790767744/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829708.994102:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829708.994105:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829708.994113:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6345 00000100:00080000:3.0:1787829708.994118:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829708.994121:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829708.994124:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829708.994133:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790767744/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829708.994134:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790768384/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829708.994179:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829708.994186:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829708.994188:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829708.994191:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6345 00000100:00080000:1.0:1787829708.994192:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829708.994193:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829708.994199:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787829708.994200:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:3.0:1787829708.994203:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829708.994207:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790767744:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829708.994215:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790768512/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829708.994222:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829708.994225:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829708.994227:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829708.994230:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6345 00000100:00080000:3.0:1787829708.994232:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829708.994235:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829708.994239:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790767872:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787829708.994240:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829708.994257:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790768640/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829708.994269:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829708.994275:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829708.994276:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829708.994281:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829708.994287:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6345 00000100:00080000:3.0:1787829708.994293:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829708.994297:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829708/real 1787829708] req@ffff9e680a2d4f00 x1874668790767872/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829708.994299:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829708.994306:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829708.994319:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790767872/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829708.994322:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790768768/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829708.994346:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829708.994352:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829708.994355:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790767872:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829708.994362:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790768000:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829708.994378:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829708.994386:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829708/real 1787829708] req@ffff9e680a2d6d00 x1874668790768000/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829708.994394:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790768000/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829708.994399:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829708.994402:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829708.994404:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790768000:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829708.994408:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790768128:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829708.994419:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829708.994424:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829708/real 1787829708] req@ffff9e680a2d5a40 x1874668790768128/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829708.994432:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790768128/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829708.994438:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829708.994439:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829708.994441:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790768128:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829708.994444:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790768256:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829708.994454:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829708.994459:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829708/real 1787829708] req@ffff9e680a2d61c0 x1874668790768256/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829708.994466:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790768256/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829708.994471:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829708.994473:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829708.994475:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790768256:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829708.994478:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790768384:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829708.994487:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829708.994493:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829708/real 1787829708] req@ffff9e680a2d5e00 x1874668790768384/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829708.994501:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790768384/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829708.994508:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829708.994509:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829708.994511:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790768384:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829708.994514:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790768512:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829708.994524:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829708.994530:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829708/real 1787829708] req@ffff9e680a2d7480 x1874668790768512/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829708.994537:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790768512/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829708.994542:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829708.994544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829708.994545:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790768512:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829708.994560:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790768640:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829708.994570:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829708.994576:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829708/real 1787829708] req@ffff9e680a2d4b40 x1874668790768640/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829708.994582:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790768640/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829708.994592:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829708.994594:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829708.994595:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790768640:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829708.994598:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790768768:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829708.994608:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829708.994627:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829708/real 1787829708] req@ffff9e680a2d7c00 x1874668790768768/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829708.994634:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790768768/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829708.994639:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829708.994642:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829708.994643:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790768768:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829709.505841:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829709.697850:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829710.529808:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829710.721894:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829711.553813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829711.745903:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829712.577849:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829712.769870:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829713.601846:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829713.793929:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829714.113813:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829714.113821:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829714.113825:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829714.113834:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6350 00000100:00080000:0.0:1787829714.113845:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829714.113853:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829714.113900:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790768896/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829714.113923:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829714.113926:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829714.113927:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829714.113930:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6350 00000100:00080000:0.0:1787829714.113933:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829714.113937:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829714.113944:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829714.113959:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790769024/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829714.113966:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829714.113969:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829714.113971:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829714.113974:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6350 00000100:00080000:0.0:1787829714.113977:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829714.113979:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829714.113982:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829714.113992:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790769152/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829714.113994:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790768896:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829714.114003:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829714.114006:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829714.114007:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829714.114010:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6350 00000100:00080000:0.0:1787829714.114012:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829714.114014:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829714.114017:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829714.114025:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790769280/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829714.114031:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829714.114033:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829714.114034:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829714.114036:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6350 00000100:00080000:0.0:1787829714.114054:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829714.114056:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829714.114059:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829714.114066:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829714.114070:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790769408/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829714.114082:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829714.114084:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829714.114086:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829714.114089:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6350 00000100:00080000:0.0:1787829714.114091:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829714.114094:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829714.114096:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829714.114105:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790769536/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829714.114111:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829714.114115:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787829714.114115:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829714/real 1787829714] req@ffff9e6803e9a940 x1874668790768896/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829714.114119:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829714.114124:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6350 00000100:00080000:0.0:1787829714.114128:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829714.114134:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829714.114139:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829714.114149:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790769664/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829714.114155:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829714.114158:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787829714.114158:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790768896/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829714.114161:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829714.114166:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6350 00000100:00080000:0.0:1787829714.114171:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829714.114177:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829714.114181:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829714.114192:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790769792/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829714.114195:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829714.114202:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829714.114203:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829714.114207:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829714.114210:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829714.114211:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790768896:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829714.114215:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6350 00000100:00080000:0.0:1787829714.114219:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829714.114223:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790769024:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829714.114224:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829714.114231:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829714.114240:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790769920/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829714.114256:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829714.114271:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829714/real 1787829714] req@ffff9e6803e9bc00 x1874668790769024/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829714.114280:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790769024/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829714.114286:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829714.114289:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829714.114291:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790769024:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829714.114309:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790769152:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829714.114320:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829714.114326:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829714/real 1787829714] req@ffff9e6803e9ad00 x1874668790769152/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829714.114334:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790769152/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829714.114339:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829714.114341:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829714.114343:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790769152:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829714.114347:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790769280:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829714.114357:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829714.114363:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829714/real 1787829714] req@ffff9e6803e98f00 x1874668790769280/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829714.114370:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790769280/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829714.114375:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829714.114377:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829714.114379:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790769280:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829714.114382:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790769408:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829714.114391:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829714.114396:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829714/real 1787829714] req@ffff9e6803e992c0 x1874668790769408/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829714.114405:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790769408/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829714.114414:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829714.114417:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829714.114421:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790769408:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829714.114427:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790769536:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829714.114447:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829714.114459:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829714/real 1787829714] req@ffff9e6803e98b40 x1874668790769536/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829714.114471:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790769536/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829714.114481:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829714.114484:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829714.114487:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790769536:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829714.114493:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790769664:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829714.114511:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829714.114520:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829714/real 1787829714] req@ffff9e6803e98780 x1874668790769664/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829714.114531:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790769664/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829714.114539:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829714.114542:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829714.114545:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790769664:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829714.114550:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790769792:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829714.114566:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829714.114575:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829714/real 1787829714] req@ffff9e6803e9a1c0 x1874668790769792/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829714.114586:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790769792/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829714.114600:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829714.114603:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829714.114606:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790769792:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829714.114612:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790769920:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829714.114627:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829714.114636:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829714/real 1787829714] req@ffff9e6803e9b840 x1874668790769920/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829714.114646:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790769920/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829714.114653:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829714.114656:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829714.114660:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790769920:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829714.625814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829714.817881:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829715.649826:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829715.841869:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829716.673770:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829716.865869:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829717.697775:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829717.889840:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829718.721778:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829718.913849:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829719.233809:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829719.233821:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829719.233827:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829719.233839:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6355 00000100:00080000:3.0:1787829719.233852:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829719.233861:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829719.233908:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790770048/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829719.233938:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829719.233942:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829719.233944:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829719.233949:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6355 00000100:00080000:3.0:1787829719.233952:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829719.233956:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829719.233968:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829719.233981:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790770176/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829719.233990:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790770048:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829719.233997:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829719.234002:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829719.234004:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829719.234008:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6355 00000100:00080000:3.0:1787829719.234012:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829719.234015:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829719.234021:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829719.234033:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790770304/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829719.234043:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829719.234048:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829719.234050:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829719.234053:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6355 00000100:00080000:3.0:1787829719.234058:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829719.234062:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829719.234067:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829719.234073:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829719.234081:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790770432/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829719.234093:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829719.234097:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829719.234099:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829719.234103:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6355 00000100:00080000:3.0:1787829719.234107:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829719.234111:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829719.234117:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829719.234127:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829719/real 1787829719] req@ffff9e680a2d4780 x1874668790770048/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829719.234140:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790770560/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829719.234153:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790770048/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829719.234153:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829719.234163:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829719.234167:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829719.234177:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6355 00000100:00080000:3.0:1787829719.234184:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829719.234190:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829719.234191:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787829719.234201:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:3.0:1787829719.234201:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829719.234209:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790770048:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829719.234219:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790770688/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829719.234230:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829719.234234:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829719.234236:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829719.234240:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6355 00000100:00100000:1.0:1787829719.234245:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790770176:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829719.234247:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829719.234256:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829719.234262:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829719.234275:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790770816/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829719.234283:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829719.234299:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829719.234300:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829719/real 1787829719] req@ffff9e680a2d43c0 x1874668790770176/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829719.234308:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829719.234312:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829719.234321:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6355 00000100:00080000:3.0:1787829719.234326:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829719.234329:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790770176/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829719.234334:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829719.234343:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829719.234348:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829719.234354:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829719.234359:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790770944/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829719.234362:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790770176:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829719.234377:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790770304:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829719.234378:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829719.234386:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829719.234388:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829719.234393:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6355 00000100:00080000:3.0:1787829719.234398:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829719.234403:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1787829719.234408:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787829719.234409:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829719.234423:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829719/real 1787829719] req@ffff9e680a2d4000 x1874668790770304/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829719.234425:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790771072/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829719.234455:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790770304/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829719.234467:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829719.234471:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829719.234475:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790770304:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829719.234482:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790770432:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829719.234500:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829719.234509:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829719/real 1787829719] req@ffff9e680a2d6580 x1874668790770432/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829719.234521:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790770432/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829719.234530:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829719.234534:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829719.234538:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790770432:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829719.234543:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790770560:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829719.234558:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829719.234567:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829719/real 1787829719] req@ffff9e680a2d70c0 x1874668790770560/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829719.234579:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790770560/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829719.234588:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829719.234591:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829719.234595:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790770560:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829719.234601:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790770688:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829719.234617:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829719.234627:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829719/real 1787829719] req@ffff9e680a2d7c00 x1874668790770688/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829719.234639:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790770688/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829719.234650:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829719.234653:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829719.234672:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790770688:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829719.234693:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790770816:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829719.234736:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829719.234748:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829719/real 1787829719] req@ffff9e680a2d4b40 x1874668790770816/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829719.234761:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790770816/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829719.234771:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829719.234774:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829719.234777:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790770816:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829719.234783:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790770944:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829719.234799:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829719.234808:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829719/real 1787829719] req@ffff9e680a2d7480 x1874668790770944/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829719.234820:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790770944/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829719.234836:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829719.234839:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829719.234843:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790770944:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829719.234848:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790771072:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829719.234865:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829719.234874:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829719/real 1787829719] req@ffff9e680a2d5e00 x1874668790771072/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829719.234885:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790771072/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829719.234894:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829719.234897:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829719.234901:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790771072:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829719.745786:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829719.937856:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829720.769825:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829720.961849:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829721.793829:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829721.986062:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829722.817802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829723.009917:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829723.841796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829724.033877:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829724.353796:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829724.353807:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829724.353811:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829724.353822:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6361 00000100:00080000:2.0:1787829724.353835:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829724.353843:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829724.353888:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790771200/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829724.353913:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829724.353916:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829724.353918:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829724.353921:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6361 00000100:00080000:2.0:1787829724.353923:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829724.353926:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829724.353937:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829724.353951:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790771328/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829724.353959:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829724.353962:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829724.353964:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829724.353969:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790771200:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829724.353970:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6361 00000100:00080000:2.0:1787829724.353976:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829724.353988:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829724.353995:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829724.354011:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790771456/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829724.354023:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829724.354027:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829724.354030:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829724.354035:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6361 00000100:00080000:2.0:1787829724.354040:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829724.354044:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829724.354051:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829724.354064:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829724.354068:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790771584/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829724.354082:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829724.354086:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829724.354089:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829724.354095:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6361 00000100:00080000:2.0:1787829724.354100:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829724.354105:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829724.354112:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829724.354125:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829724/real 1787829724] req@ffff9e6804420b40 x1874668790771200/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829724.354128:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790771712/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829724.354144:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829724.354154:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829724.354156:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829724.354162:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6361 00000100:00100000:1.0:1787829724.354167:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790771200/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829724.354169:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829724.354179:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829724.354189:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829724.354195:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829724.354202:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829724.354207:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790771840/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829724.354209:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790771200:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829724.354230:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829724.354233:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829724.354235:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829724.354239:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6361 00000100:00080000:2.0:1787829724.354242:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829724.354246:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790771328:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829724.354254:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829724.354261:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829724.354274:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829724.354280:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790771968/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829724.354292:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829724/real 1787829724] req@ffff9e68044230c0 x1874668790771328/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829724.354296:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829724.354306:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829724.354311:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829724.354317:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6361 00000100:00100000:1.0:1787829724.354321:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790771328/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829724.354322:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829724.354330:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829724.354338:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829724.354341:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829724.354347:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829724.354351:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790772096/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829724.354356:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790771328:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829724.354369:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790771456:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829724.354373:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829724.354379:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829724.354382:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829724.354388:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6361 00000100:00080000:2.0:1787829724.354394:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829724.354397:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829724.354402:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829724.354411:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829724.354412:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829724/real 1787829724] req@ffff9e68044212c0 x1874668790771456/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829724.354426:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790771456/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829724.354434:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829724.354440:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829724.354441:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790772224/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829724.354447:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790771456:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829724.354462:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790771584:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829724.354484:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829724.354494:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829724/real 1787829724] req@ffff9e6804421e00 x1874668790771584/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829724.354506:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790771584/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829724.354515:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829724.354519:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829724.354522:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790771584:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829724.354529:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790771712:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829724.354547:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829724.354558:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829724/real 1787829724] req@ffff9e6804423840 x1874668790771712/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829724.354569:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790771712/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829724.354577:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829724.354581:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829724.354584:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790771712:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829724.354590:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790771840:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829724.354605:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829724.354615:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829724/real 1787829724] req@ffff9e68044221c0 x1874668790771840/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829724.354640:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790771840/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829724.354650:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829724.354653:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829724.354657:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790771840:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829724.354675:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790771968:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829724.354720:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829724.354732:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829724/real 1787829724] req@ffff9e6804420780 x1874668790771968/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829724.354744:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790771968/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829724.354751:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829724.354755:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829724.354758:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790771968:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829724.354764:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790772096:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829724.354781:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829724.354790:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829724/real 1787829724] req@ffff9e6804422940 x1874668790772096/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829724.354800:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790772096/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829724.354813:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829724.354816:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829724.354820:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790772096:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829724.354826:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790772224:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829724.354842:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829724.354851:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829724/real 1787829724] req@ffff9e6804421680 x1874668790772224/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829724.354861:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790772224/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829724.354868:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829724.354871:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829724.354874:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790772224:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829724.865836:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829725.057888:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829725.889796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829726.081858:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829726.913821:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829727.106101:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829727.937828:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829728.129879:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829728.961968:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829729.153907:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829729.473811:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829729.473824:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829729.473831:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829729.473843:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6366 00000100:00080000:3.0:1787829729.473856:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829729.473865:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829729.473913:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790772352/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829729.473951:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829729.473955:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829729.473957:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829729.473962:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6366 00000100:00080000:3.0:1787829729.473966:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829729.473970:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829729.473982:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829729.473996:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790772480/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829729.474002:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790772352:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829729.474012:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829729.474017:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829729.474019:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829729.474025:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6366 00000100:00080000:3.0:1787829729.474029:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829729.474033:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829729.474038:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829729.474051:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790772608/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829729.474069:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829729.474073:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829729.474075:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829729.474078:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6366 00000800:00000100:1.0:1787829729.474082:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829729.474085:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829729.474092:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829729.474098:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829729.474110:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790772736/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829729.474120:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829729.474125:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829729.474129:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829729.474130:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829729/real 1787829729] req@ffff9e680a2d61c0 x1874668790772352/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829729.474138:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6366 00000100:00080000:3.0:1787829729.474145:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829729.474150:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829729.474153:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790772352/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829729.474157:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829729.474171:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790772864/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829729.474181:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829729.474186:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829729.474188:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829729.474195:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790772352:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829729.474195:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829729.474198:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829729.474205:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6366 00000100:00080000:3.0:1787829729.474209:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829729.474213:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829729.474218:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829729.474222:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790772480:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829729.474234:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790772992/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829729.474244:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829729.474250:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829729.474255:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787829729.474259:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829729/real 1787829729] req@ffff9e680a2d5a40 x1874668790772480/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829729.474260:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829729.474269:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6366 00000100:00100000:1.0:1787829729.474275:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790772480/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829729.474276:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829729.474285:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787829729.474286:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829729.474292:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:3.0:1787829729.474293:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829729.474295:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790772480:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829729.474303:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790772608:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829729.474311:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790773120/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829729.474319:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829729.474328:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829729.474329:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829729/real 1787829729] req@ffff9e680a2d6d00 x1874668790772608/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829729.474336:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829729.474341:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829729.474345:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790772608/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829729.474348:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6366 00000100:00080000:3.0:1787829729.474355:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829729.474357:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829729.474362:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829729.474366:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829729.474367:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790772608:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787829729.474376:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829729.474377:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790772736:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829729.474393:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790773248/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829729.474405:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829729.474417:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829729.474422:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829729.474423:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829729/real 1787829729] req@ffff9e680a2d4f00 x1874668790772736/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829729.474425:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829729.474433:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6366 00000100:00080000:3.0:1787829729.474440:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829729.474445:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829729.474447:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790772736/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829729.474452:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829729.474457:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829729.474460:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829729.474464:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790772736:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829729.474468:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790773376/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829729.474472:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790772864:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829729.474504:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829729.474517:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829729/real 1787829729] req@ffff9e680a2d52c0 x1874668790772864/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829729.474530:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790772864/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829729.474537:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829729.474539:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829729.474542:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790772864:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829729.474558:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790772992:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829729.474573:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829729.474581:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829729/real 1787829729] req@ffff9e680a2d61c0 x1874668790772992/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829729.474611:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790772992/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829729.474622:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829729.474626:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829729.474630:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790772992:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829729.474636:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790773120:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829729.474655:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829729.474664:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829729/real 1787829729] req@ffff9e680a2d5e00 x1874668790773120/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829729.474672:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790773120/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829729.474677:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829729.474679:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829729.474691:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790773120:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829729.474717:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790773248:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829729.474738:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829729.474749:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829729/real 1787829729] req@ffff9e680a2d7480 x1874668790773248/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829729.474761:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790773248/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829729.474777:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829729.474781:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829729.474785:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790773248:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829729.474791:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790773376:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829729.474817:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829729.474829:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829729/real 1787829729] req@ffff9e680a2d4b40 x1874668790773376/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829729.474841:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790773376/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829729.474850:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829729.474853:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829729.474857:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790773376:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829729.985798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829730.177875:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829731.009839:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829731.201860:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829732.033786:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829732.225839:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829733.057774:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829733.249895:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829734.081782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829734.273894:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829734.593897:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829734.593908:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829734.593913:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829734.593925:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6371 00000100:00080000:1.0:1787829734.593938:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829734.593948:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829734.593999:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790773504/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829734.594033:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829734.594036:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829734.594039:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829734.594051:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6371 00000100:00080000:1.0:1787829734.594057:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829734.594064:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829734.594067:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790773504:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787829734.594078:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829734.594099:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790773632/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829734.594109:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829734.594113:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829734.594114:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829734.594118:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6371 00000100:00080000:1.0:1787829734.594123:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829734.594127:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829734.594132:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829734.594133:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829734.594164:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790773760/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829734.594175:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829734.594178:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829734.594181:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829734.594186:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829734/real 1787829734] req@ffff9e6831aaa580 x1874668790773504/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829734.594187:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6371 00000100:00080000:1.0:1787829734.594201:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829734.594216:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829734.594222:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829734.594233:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790773504/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829734.594234:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790773888/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829734.594279:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829734.594285:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829734.594288:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829734.594289:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829734.594296:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6371 00000100:00080000:0.0:1787829734.594301:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829734.594302:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829734.594310:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829734.594311:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790773504:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787829734.594318:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829734.594333:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790774016/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829734.594341:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829734.594343:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829734.594345:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829734.594350:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6371 00000100:00080000:1.0:1787829734.594355:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829734.594360:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829734.594361:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790773632:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787829734.594367:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829734.594382:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790774144/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829734.594398:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829734.594404:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000800:00000100:0.0:1787829734.594405:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829734.594413:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829734.594421:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6371 00000100:00080000:1.0:1787829734.594426:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829734.594431:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829734.594436:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829734.594442:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829734/real 1787829734] req@ffff9e6831aa8f00 x1874668790773632/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829734.594452:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790774272/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829734.594459:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790773632/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829734.594464:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829734.594480:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829734.594485:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829734.594495:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6371 00000100:00080000:0.0:1787829734.594496:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829734.594502:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829734.594507:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829734.594514:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829734.594518:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790773632:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787829734.594524:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829734.594531:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790773760:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829734.594538:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790774400/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829734.594560:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829734.594564:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829734.594566:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829734.594569:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829734.594575:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6371 00000100:00080000:1.0:1787829734.594581:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829734.594587:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829734.594589:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829734/real 1787829734] req@ffff9e6831aa9e00 x1874668790773760/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829734.594594:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829734.594609:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790773760/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829734.594616:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790774528/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829734.594634:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829734.594643:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829734.594656:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790773760:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829734.594670:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790773888:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829734.594716:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829734.594733:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829734/real 1787829734] req@ffff9e6831aa8b40 x1874668790773888/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829734.594747:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790773888/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829734.594757:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829734.594762:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829734.594767:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790773888:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829734.594776:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790774016:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829734.594798:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829734.594807:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829734/real 1787829734] req@ffff9e6831aab480 x1874668790774016/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829734.594814:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790774016/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829734.594820:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829734.594822:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829734.594824:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790774016:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829734.594841:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790774144:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829734.594861:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829734.594872:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829734/real 1787829734] req@ffff9e6831aaad00 x1874668790774144/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829734.594885:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790774144/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829734.594903:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829734.594905:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829734.594908:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790774144:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829734.594912:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790774272:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829734.594928:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829734.594936:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829734/real 1787829734] req@ffff9e6831aaa1c0 x1874668790774272/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829734.594943:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790774272/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829734.594949:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829734.594951:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829734.594953:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790774272:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829734.594957:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790774400:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829734.594967:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829734.594972:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829734/real 1787829734] req@ffff9e6831aa8780 x1874668790774400/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829734.594979:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790774400/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829734.594984:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829734.594986:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829734.594988:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790774400:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829734.594992:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790774528:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829734.595002:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829734.595007:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829734/real 1787829734] req@ffff9e6831aaa940 x1874668790774528/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829734.595015:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790774528/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829734.595020:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829734.595021:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829734.595023:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790774528:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829735.105859:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829735.297841:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829736.129799:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829736.321834:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829737.153842:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829737.345850:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829738.177830:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829738.369926:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829739.201774:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829739.393892:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829739.713824:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829739.713834:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829739.713839:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829739.713850:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6376 00000100:00080000:3.0:1787829739.713862:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829739.713869:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829739.713916:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790774656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829739.713950:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829739.713955:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829739.713958:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829739.713967:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6376 00000100:00080000:3.0:1787829739.713971:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829739.713974:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829739.713986:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829739.713994:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790774656:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829739.714003:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790774784/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829739.714015:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829739.714017:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829739.714020:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829739.714024:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6376 00000100:00080000:3.0:1787829739.714026:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829739.714029:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829739.714034:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829739.714049:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790774912/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829739.714061:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829739.714063:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829739.714069:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829739.714072:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829739.714079:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6376 00000100:00080000:3.0:1787829739.714087:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829739.714091:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829739.714098:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829739.714101:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829739/real 1787829739] req@ffff9e680a2d7c00 x1874668790774656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829739.714116:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790775040/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829739.714125:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790774656/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829739.714128:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829739.714140:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829739.714143:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829739.714154:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6376 00000100:00080000:3.0:1787829739.714160:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829739.714165:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829739.714171:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829739.714172:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829739.714187:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790775168/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829739.714188:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829739.714205:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790774656:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829739.714208:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829739.714215:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829739.714225:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829739.714229:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6376 00000100:00080000:3.0:1787829739.714236:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829739.714237:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790774784:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829739.714247:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829739.714252:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829739.714268:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790775296/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829739.714278:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829739.714288:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829739.714292:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829739.714294:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829739.714300:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6376 00000100:00100000:0.0:1787829739.714301:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829739/real 1787829739] req@ffff9e680a2d70c0 x1874668790774784/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829739.714308:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829739.714320:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829739.714329:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829739.714330:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790774784/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829739.714349:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829739.714351:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790775424/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829739.714356:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829739.714370:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790774784:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829739.714391:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829739.714399:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790774912:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829739.714399:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829739.714407:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829739.714417:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6376 00000100:00080000:3.0:1787829739.714421:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829739.714425:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787829739.714430:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787829739.714430:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829739.714446:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829739/real 1787829739] req@ffff9e680a2d6580 x1874668790774912/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829739.714451:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790775552/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829739.714504:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829739.714506:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790774912/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829739.714527:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829739.714530:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829739.714531:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829739.714544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829739.714547:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6376 00000100:00100000:0.0:1787829739.714554:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790774912:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829739.714556:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829739.714564:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829739.714568:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790775040:192.168.203.109@tcp:8:kworker.0 00000008:00100000:3.0:1787829739.714573:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829739.714590:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790775680/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829739.714601:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829739.714616:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829739/real 1787829739] req@ffff9e680a2d4000 x1874668790775040/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829739.714626:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790775040/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829739.714636:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829739.714639:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829739.714642:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790775040:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829739.714647:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790775168:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829739.714657:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829739.714663:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829739/real 1787829739] req@ffff9e680a2d43c0 x1874668790775168/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829739.714670:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790775168/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829739.714675:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829739.714677:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829739.714679:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790775168:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829739.714715:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790775296:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829739.714729:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829739.714736:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829739/real 1787829739] req@ffff9e680a2d4780 x1874668790775296/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829739.714743:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790775296/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829739.714750:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829739.714752:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829739.714755:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790775296:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829739.714759:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790775424:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829739.714770:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829739.714775:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829739/real 1787829739] req@ffff9e680a2d7c00 x1874668790775424/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829739.714783:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790775424/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829739.714787:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829739.714789:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829739.714791:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790775424:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829739.714795:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790775552:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829739.714804:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829739.714809:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829739/real 1787829739] req@ffff9e680a2d4b40 x1874668790775552/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829739.714815:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790775552/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829739.714820:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829739.714822:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829739.714824:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790775552:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829739.714828:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790775680:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829739.714837:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829739.714843:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829739/real 1787829739] req@ffff9e680a2d7480 x1874668790775680/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829739.714850:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790775680/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829739.714855:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829739.714857:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829739.714859:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790775680:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829740.225841:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829740.417893:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829741.249817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829741.441894:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829742.273806:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829742.465865:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829743.297804:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829743.489879:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829744.321827:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829744.513908:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829744.833777:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829744.833788:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829744.833793:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829744.833804:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6381 00000100:00080000:3.0:1787829744.833817:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829744.833824:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829744.833869:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790775808/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829744.833901:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829744.833904:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829744.833905:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829744.833909:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6381 00000100:00080000:3.0:1787829744.833911:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829744.833914:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829744.833929:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829744.833940:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790775936/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829744.833947:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790775808:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829744.833950:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829744.833957:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829744.833958:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829744.833962:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6381 00000100:00080000:3.0:1787829744.833964:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829744.833968:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829744.833971:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829744.833980:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790776064/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829744.833986:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829744.833990:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829744.833991:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829744.833994:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6381 00000100:00080000:3.0:1787829744.833997:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829744.834001:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829744.834007:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829744.834019:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790776192/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829744.834030:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829744.834035:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829744.834037:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829744.834041:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829744.834046:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6381 00000100:00080000:3.0:1787829744.834051:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829744.834055:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829744.834060:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829744.834071:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790776320/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829744.834079:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829744.834082:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787829744.834085:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829744/real 1787829744] req@ffff9e680a2d5e00 x1874668790775808/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829744.834085:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829744.834093:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6381 00000100:00080000:3.0:1787829744.834098:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829744.834104:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829744.834108:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829744.834113:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790775808/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829744.834118:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790776448/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829744.834130:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829744.834136:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829744.834136:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829744.834145:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790775808:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829744.834145:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829744.834150:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829744.834157:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6381 00000100:00080000:3.0:1787829744.834162:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829744.834167:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829744.834173:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790775936:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787829744.834174:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829744.834192:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790776576/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829744.834200:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829744.834202:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829744.834207:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829744.834208:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829744.834214:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6381 00000100:00100000:0.0:1787829744.834215:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829744/real 1787829744] req@ffff9e680a2d61c0 x1874668790775936/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829744.834217:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829744.834225:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829744.834231:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829744.834232:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790775936/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829744.834243:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790776704/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829744.834254:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829744.834258:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829744.834261:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829744.834265:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787829744.834268:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790775936:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829744.834270:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829744.834275:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790776064:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829744.834280:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6381 00000100:00080000:3.0:1787829744.834285:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829744.834290:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787829744.834292:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787829744.834301:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829744.834304:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829744/real 1787829744] req@ffff9e680a2d52c0 x1874668790776064/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829744.834313:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790776064/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829744.834319:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790776832/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829744.834322:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829744.834331:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829744.834334:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790776064:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829744.834339:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790776192:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829744.834354:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829744.834362:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829744/real 1787829744] req@ffff9e680a2d4f00 x1874668790776192/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829744.834370:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790776192/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829744.834375:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829744.834377:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829744.834380:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790776192:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829744.834383:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790776320:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829744.834392:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829744.834398:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829744/real 1787829744] req@ffff9e680a2d6d00 x1874668790776320/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829744.834404:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790776320/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829744.834409:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829744.834411:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829744.834413:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790776320:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829744.834416:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790776448:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829744.834433:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829744.834444:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829744/real 1787829744] req@ffff9e680a2d5a40 x1874668790776448/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829744.834456:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790776448/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829744.834466:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829744.834470:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829744.834473:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790776448:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829744.834510:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790776576:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829744.834533:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829744.834545:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829744/real 1787829744] req@ffff9e680a2d5e00 x1874668790776576/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829744.834554:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790776576/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829744.834560:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829744.834562:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829744.834564:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790776576:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829744.834568:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790776704:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829744.834581:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829744.834587:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829744/real 1787829744] req@ffff9e680a2d7480 x1874668790776704/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829744.834594:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790776704/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829744.834599:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829744.834601:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829744.834603:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790776704:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829744.834606:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790776832:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829744.834616:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829744.834621:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829744/real 1787829744] req@ffff9e680a2d4b40 x1874668790776832/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829744.834628:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790776832/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829744.834634:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829744.834636:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829744.834638:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790776832:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829745.345832:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829745.537861:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829746.369808:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829746.561835:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829747.393802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829747.585862:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829748.417813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829748.609952:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829749.441870:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829749.633868:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829749.953787:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829749.953795:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829749.953799:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829749.953807:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6386 00000100:00080000:0.0:1787829749.953815:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829749.953822:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829749.953853:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790776960/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829749.953886:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829749.953889:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829749.953890:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829749.953893:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6386 00000100:00080000:0.0:1787829749.953895:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829749.953898:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829749.953905:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829749.953922:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790777088/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829749.953925:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790776960:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829749.953941:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829749.953944:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829749.953949:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829749.953952:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6386 00000100:00080000:0.0:1787829749.953955:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829749.953957:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829749.953961:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829749.953970:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790777216/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829749.953976:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829749.953979:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829749.953980:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829749.953982:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6386 00000100:00080000:0.0:1787829749.953991:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829749.953994:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829749.953997:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829749.954002:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829749.954007:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790777344/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829749.954014:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829749.954016:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829749.954018:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829749.954021:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6386 00000100:00080000:0.0:1787829749.954023:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829749.954025:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829749.954028:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829749.954037:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790777472/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829749.954043:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829749.954045:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829749/real 1787829749] req@ffff9e6803e9a580 x1874668790776960/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829749.954046:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829749.954049:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829749.954056:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6386 00000100:00080000:0.0:1787829749.954060:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829749.954066:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829749.954072:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829749.954082:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790777600/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829749.954084:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790776960/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829749.954103:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829749.954106:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829749.954107:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829749.954108:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829749.954112:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829749.954115:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6386 00000100:00080000:0.0:1787829749.954119:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829749.954119:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790776960:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829749.954124:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829749.954128:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829749.954135:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790777728/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829749.954142:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829749.954145:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829749.954147:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829749.954150:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6386 00000100:00080000:0.0:1787829749.954152:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829749.954154:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790777088:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829749.954158:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829749.954163:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829749.954170:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790777856/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829749.954177:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829749.954180:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829749.954181:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829749.954184:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6386 00000800:00000100:1.0:1787829749.954188:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829749.954189:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829749.954194:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829749.954198:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829749.954203:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829749/real 1787829749] req@ffff9e6803e99680 x1874668790777088/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829749.954207:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790777984/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829749.954228:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790777088/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829749.954237:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829749.954240:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829749.954244:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790777088:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829749.954249:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790777216:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829749.954270:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829749.954277:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829749/real 1787829749] req@ffff9e6803e99e00 x1874668790777216/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829749.954284:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790777216/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829749.954294:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829749.954296:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829749.954298:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790777216:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829749.954302:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790777344:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829749.954313:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829749.954319:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829749/real 1787829749] req@ffff9e6803e9b480 x1874668790777344/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829749.954326:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790777344/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829749.954331:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829749.954333:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829749.954336:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790777344:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829749.954341:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790777472:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829749.954359:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829749.954370:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829749/real 1787829749] req@ffff9e6803e9b840 x1874668790777472/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829749.954383:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790777472/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829749.954391:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829749.954393:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829749.954396:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790777472:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829749.954401:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790777600:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829749.954415:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829749.954423:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829749/real 1787829749] req@ffff9e6803e9a1c0 x1874668790777600/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829749.954430:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790777600/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829749.954436:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829749.954438:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829749.954456:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790777600:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829749.954470:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790777728:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829749.954485:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829749.954492:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829749/real 1787829749] req@ffff9e6803e98780 x1874668790777728/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829749.954500:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790777728/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829749.954505:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829749.954507:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829749.954509:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790777728:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829749.954513:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790777856:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829749.954522:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829749.954532:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829749/real 1787829749] req@ffff9e6803e98b40 x1874668790777856/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829749.954544:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790777856/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829749.954552:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829749.954556:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829749.954559:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790777856:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829749.954565:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790777984:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829749.954585:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829749.954593:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829749/real 1787829749] req@ffff9e6803e992c0 x1874668790777984/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829749.954600:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790777984/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829749.954605:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829749.954608:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829749.954609:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790777984:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829750.465859:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829750.657882:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829751.489787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829751.681886:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829752.513800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829752.705866:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829753.537794:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829753.729852:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829754.561836:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829754.753921:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829755.073837:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829755.073846:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829755.073849:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829755.073858:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6391 00000100:00080000:3.0:1787829755.073867:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829755.073873:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829755.073908:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790778112/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829755.073933:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829755.073936:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829755.073938:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829755.073940:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6391 00000100:00080000:3.0:1787829755.073943:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829755.073945:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829755.073953:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829755.073962:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790778240/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829755.073969:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829755.073972:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829755.073973:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829755.073976:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6391 00000100:00080000:3.0:1787829755.073977:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829755.073980:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829755.073983:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829755.073990:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790778112:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829755.073991:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790778368/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829755.074007:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829755.074009:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829755.074011:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829755.074013:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6391 00000100:00080000:3.0:1787829755.074015:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829755.074018:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829755.074022:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829755.074029:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790778496/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829755.074043:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829755.074045:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829755.074046:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829755.074049:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6391 00000100:00080000:3.0:1787829755.074052:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829755.074054:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829755.074057:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829755.074068:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829755.074068:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790778624/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829755.074077:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829755.074080:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829755.074082:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829755.074084:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6391 00000100:00080000:3.0:1787829755.074087:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829755.074090:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829755.074092:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829755.074100:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790778752/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829755.074106:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829755.074108:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829755.074110:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829755.074113:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6391 00000100:00080000:3.0:1787829755.074115:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829755.074119:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829755.074122:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829755.074130:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829755/real 1787829755] req@ffff9e680a2d7c00 x1874668790778112/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829755.074130:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790778880/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829755.074158:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829755.074164:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829755.074167:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829755.074172:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6391 00000100:00080000:3.0:1787829755.074175:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829755.074178:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829755.074181:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829755.074188:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790778112/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829755.074190:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790779008/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829755.074218:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829755.074219:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829755.074221:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829755.074225:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829755.074228:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829755.074231:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6391 00000100:00080000:3.0:1787829755.074235:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829755.074236:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790778112:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829755.074240:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829755.074245:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829755.074248:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790778240:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829755.074255:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790779136/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829755.074280:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829755.074297:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829755/real 1787829755] req@ffff9e680a2d4780 x1874668790778240/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829755.074311:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790778240/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829755.074320:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829755.074324:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829755.074326:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790778240:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829755.074347:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790778368:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829755.074364:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829755.074373:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829755/real 1787829755] req@ffff9e680a2d43c0 x1874668790778368/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829755.074381:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790778368/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829755.074390:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829755.074393:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829755.074395:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790778368:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829755.074399:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790778496:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829755.074428:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829755.074436:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829755/real 1787829755] req@ffff9e680a2d4000 x1874668790778496/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829755.074444:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790778496/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829755.074451:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829755.074454:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829755.074458:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790778496:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829755.074464:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790778624:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829755.074484:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829755.074495:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829755/real 1787829755] req@ffff9e680a2d6580 x1874668790778624/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829755.074508:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790778624/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829755.074514:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829755.074516:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829755.074519:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790778624:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829755.074523:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790778752:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829755.074538:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829755.074545:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829755/real 1787829755] req@ffff9e680a2d70c0 x1874668790778752/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829755.074553:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790778752/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829755.074559:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829755.074561:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829755.074563:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790778752:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829755.074566:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790778880:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829755.074583:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829755.074593:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829755/real 1787829755] req@ffff9e680a2d4b40 x1874668790778880/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829755.074605:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790778880/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829755.074614:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829755.074617:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829755.074620:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790778880:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829755.074627:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790779008:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829755.074647:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829755.074658:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829755/real 1787829755] req@ffff9e680a2d7480 x1874668790779008/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829755.074666:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790779008/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829755.074672:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829755.074674:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829755.074677:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790779008:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829755.074683:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790779136:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829755.074729:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829755.074741:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829755/real 1787829755] req@ffff9e680a2d5e00 x1874668790779136/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829755.074749:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790779136/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829755.074755:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829755.074757:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829755.074759:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790779136:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829755.585779:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829755.777878:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829756.609814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829756.801885:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829757.633796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829757.825925:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829758.657861:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829758.849890:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829759.681830:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829759.873895:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829760.193806:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829760.193816:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829760.193820:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829760.193831:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6396 00000100:00080000:1.0:1787829760.193842:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829760.193850:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829760.193888:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790779264/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829760.193945:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829760.193950:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829760.193952:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829760.193957:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6396 00000100:00080000:1.0:1787829760.193963:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829760.193968:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829760.193979:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829760.193987:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790779264:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787829760.193996:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790779392/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829760.194010:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829760.194014:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829760.194016:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829760.194021:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6396 00000100:00080000:1.0:1787829760.194040:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829760.194046:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787829760.194051:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1787829760.194055:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829760.194073:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790779520/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829760.194089:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829760/real 1787829760] req@ffff9e6831aa9680 x1874668790779264/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829760.194093:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829760.194105:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829760.194108:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829760.194112:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790779264/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829760.194115:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6396 00000100:00080000:1.0:1787829760.194126:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829760.194128:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829760.194134:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829760.194134:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829760.194141:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790779264:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787829760.194146:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829760.194164:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790779392:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829760.194166:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790779648/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829760.194181:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829760.194185:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829760.194190:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829760.194193:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829760.194197:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829760/real 1787829760] req@ffff9e6831aabc00 x1874668790779392/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829760.194201:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6396 00000100:00080000:1.0:1787829760.194211:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829760.194215:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790779392/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829760.194221:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787829760.194227:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787829760.194230:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829760.194232:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829760.194238:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790779392:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829760.194250:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790779520:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829760.194251:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790779776/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829760.194265:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829760.194269:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829760.194271:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829760.194277:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6396 00000800:00000100:0.0:1787829760.194279:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829760.194286:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829760.194294:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829760.194298:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829760/real 1787829760] req@ffff9e6831aa9a40 x1874668790779520/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829760.194302:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829760.194324:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790779904/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829760.194325:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790779520/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829760.194360:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829760.194361:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829760.194385:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829760.194388:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829760.194389:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829760.194396:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790779520:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829760.194400:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6396 00000100:00080000:1.0:1787829760.194407:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829760.194412:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829760.194415:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790779648:192.168.203.109@tcp:8:kworker.0 00000008:00100000:1.0:1787829760.194422:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829760.194440:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829760.194440:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790780032/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829760.194452:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829760/real 1787829760] req@ffff9e6831aa8000 x1874668790779648/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829760.194456:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829760.194466:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787829760.194470:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790779648/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829760.194470:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829760.194481:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6396 00000100:00080000:0.0:1787829760.194483:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829760.194489:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829760.194490:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829760.194492:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790779648:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787829760.194499:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829760.194500:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790779776:192.168.203.109@tcp:8:kworker.0 00000008:00100000:1.0:1787829760.194511:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829760.194517:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829760.194525:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829760/real 1787829760] req@ffff9e6831aa9680 x1874668790779776/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829760.194532:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790779776/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829760.194540:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787829760.194540:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790780160/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829760.194545:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829760.194547:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790779776:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829760.194556:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790779904:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829760.194571:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829760.194573:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829760.194575:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829760.194578:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6396 00000100:00080000:1.0:1787829760.194582:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829760.194585:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829760.194590:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829760.194593:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829760.194605:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790780288/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829760.194606:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829760/real 1787829760] req@ffff9e6831aaa940 x1874668790779904/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829760.194641:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790779904/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829760.194650:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829760.194653:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829760.194655:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790779904:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829760.194666:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790780032:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829760.194677:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829760.194683:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829760/real 1787829760] req@ffff9e6831aa8780 x1874668790780032/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829760.194690:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790780032/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829760.194713:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829760.194715:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829760.194717:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790780032:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829760.194721:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790780160:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829760.194733:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829760.194739:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829760/real 1787829760] req@ffff9e6831aaa1c0 x1874668790780160/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829760.194746:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790780160/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829760.194751:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829760.194753:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829760.194756:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790780160:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829760.194759:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790780288:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829760.194769:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829760.194774:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829760/real 1787829760] req@ffff9e6831aaad00 x1874668790780288/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829760.194781:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790780288/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829760.194786:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829760.194787:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829760.194789:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790780288:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829760.705814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829760.897879:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829761.729807:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829761.921906:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829762.753785:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829762.945882:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829763.777816:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829763.969876:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829764.801767:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829764.993844:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829765.313804:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829765.313813:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829765.313816:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829765.313824:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6402 00000100:00080000:2.0:1787829765.313833:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829765.313839:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829765.313873:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790780416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829765.313895:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829765.313897:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829765.313899:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829765.313902:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6402 00000100:00080000:2.0:1787829765.313904:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829765.313907:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829765.313914:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829765.313923:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790780544/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829765.313931:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829765.313935:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787829765.313936:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790780416:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829765.313937:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829765.313942:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6402 00000100:00080000:2.0:1787829765.313947:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829765.313950:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829765.313954:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829765.313963:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790780672/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829765.313969:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829765.313971:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829765.313973:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829765.313976:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6402 00000100:00080000:2.0:1787829765.313978:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829765.313981:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829765.313984:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829765.313991:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790780800/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829765.313996:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829765.313999:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829765.314000:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829765.314002:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6402 00000100:00080000:2.0:1787829765.314005:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829765.314007:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829765.314009:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829765.314018:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790780928/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829765.314024:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829765.314027:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829765.314029:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829765.314032:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6402 00000100:00080000:2.0:1787829765.314034:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829765.314037:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829765.314041:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829765.314043:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790780544:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829765.314052:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790781056/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829765.314062:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829765.314064:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829765.314065:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829765.314068:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6402 00000100:00080000:2.0:1787829765.314071:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829765.314073:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790780672:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829765.314076:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829765.314082:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829765.314091:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790781184/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829765.314097:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790780800:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787829765.314099:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829765.314104:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829765.314106:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829765.314110:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6402 00000100:00080000:2.0:1787829765.314114:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829765.314118:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790780928:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787829765.314119:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829765.314133:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829765.314137:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790781056:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829765.314144:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790781312/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829765.314153:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829765.314157:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790781184:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829765.314158:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829765.314162:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829765.314167:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6402 00000100:00080000:2.0:1787829765.314170:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829765.314173:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829765.314181:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790781312:192.168.203.109@tcp:8:kworker.0 00000008:00100000:2.0:1787829765.314181:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829765.314197:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790781440/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829765.314210:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790781440:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829765.825807:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829766.017945:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829766.849763:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829767.041861:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829767.873812:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829768.065956:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787829768.386096:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:1.0:1787829768.386176:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787829768.386200:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1787829768.386298:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829765/real 1787829768] req@ffff9e6804422d00 x1874668790780416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829790 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829768.386344:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790780416/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829790 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829768.386359:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829768.386365:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829768.386370:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790780416:192.168.203.109@tcp:250:kworker.0 00000800:00000100:1.0:1787829768.387988:0:2386:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:1.0:1787829768.388023:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829768.388038:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829768.388048:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829768.388060:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829768.388069:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1787829768.388078:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829765/real 1787829768] req@ffff9e6804423840 x1874668790781440/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787829768.388082:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829768.388095:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1787829768.388132:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790781440/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829768.388144:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829768.388149:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829768.388154:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790781440:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829768.388163:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829765/real 1787829768] req@ffff9e68044221c0 x1874668790781312/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829768.388174:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790781312/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829768.388181:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829768.388185:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829768.388189:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790781312:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829768.388197:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829765/real 1787829768] req@ffff9e6804422940 x1874668790781056/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829768.388207:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790781056/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829768.388215:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829768.388218:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829768.388221:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790781056:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829768.388229:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829765/real 1787829768] req@ffff9e6804420780 x1874668790781184/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829768.388238:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790781184/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829768.388245:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829768.388249:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829768.388252:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790781184:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829768.388260:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829765/real 1787829768] req@ffff9e6804423480 x1874668790780544/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829768.388275:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790780544/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829768.388284:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829768.388288:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829768.388291:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790780544:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829768.388300:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829765/real 1787829768] req@ffff9e6804420f00 x1874668790780672/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829768.388329:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790780672/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829768.388338:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829768.388343:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829768.388346:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790780672:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829768.388354:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829765/real 1787829768] req@ffff9e6804422580 x1874668790780800/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829768.388367:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790780800/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829768.388376:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829768.388380:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829768.388383:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790780800:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829768.388391:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829765/real 1787829768] req@ffff9e6804421680 x1874668790780928/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829768.388401:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790780928/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829768.388407:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829768.388409:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829768.388413:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790780928:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829768.897815:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829769.091052:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829769.921863:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829770.113910:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829770.433960:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829770.433972:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829770.433976:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829770.433989:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6407 00000100:00080000:1.0:1787829770.434001:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829770.434008:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829770.434053:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790781568/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829770.434091:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829770.434096:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829770.434098:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829770.434103:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6407 00000100:00080000:1.0:1787829770.434108:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829770.434113:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829770.434124:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829770.434137:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790781696/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829770.434145:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829770.434149:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829770.434151:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829770.434165:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6407 00000100:00080000:1.0:1787829770.434170:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829770.434177:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829770.434179:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790781568:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787829770.434184:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829770.434200:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790781824/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829770.434212:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829770.434217:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829770.434219:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829770.434223:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6407 00000100:00080000:1.0:1787829770.434228:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829770.434233:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829770.434239:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829770.434253:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790781952/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829770.434264:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829770.434267:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829770.434270:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829770.434275:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829770.434278:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6407 00000100:00080000:1.0:1787829770.434284:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829770.434289:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829770.434295:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829770.434323:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790782080/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829770.434334:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829770.434338:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829770.434340:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829770.434347:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6407 00000100:00100000:0.0:1787829770.434349:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829770/real 1787829770] req@ffff9e6831aab480 x1874668790781568/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829770.434351:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829770.434359:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829770.434369:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829770.434384:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790781568/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829770.434398:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829770.434404:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829770.434408:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790782208/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829770.434413:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790781568:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829770.434431:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829770.434432:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790781696:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829770.434440:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829770.434444:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829770.434450:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6407 00000100:00080000:1.0:1787829770.434456:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829770.434461:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829770.434467:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829770.434471:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829770.434487:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790782336/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829770.434493:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829770/real 1787829770] req@ffff9e6831aa8b40 x1874668790781696/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829770.434504:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829770.434513:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829770.434516:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829770.434519:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790781696/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829770.434524:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6407 00000100:00080000:1.0:1787829770.434532:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829770.434539:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829770.434540:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787829770.434549:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:1.0:1787829770.434551:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829770.434556:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790781696:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829770.434569:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790782464/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829770.434580:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829770.434585:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829770.434587:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829770.434592:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790781824:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829770.434596:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6407 00000100:00080000:1.0:1787829770.434604:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829770.434608:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829770.434614:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829770.434629:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829770.434631:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790782592/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829770.434648:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829770/real 1787829770] req@ffff9e6831aa9e00 x1874668790781824/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829770.434665:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790781824/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829770.434674:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829770.434678:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829770.434681:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790781824:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829770.434689:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790781952:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829770.434762:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829770.434775:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829770/real 1787829770] req@ffff9e6831aa8f00 x1874668790781952/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829770.434787:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790781952/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829770.434796:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829770.434799:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829770.434802:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790781952:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829770.434809:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790782080:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829770.434838:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829770.434851:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829770/real 1787829770] req@ffff9e6831aaa580 x1874668790782080/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829770.434864:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790782080/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829770.434881:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829770.434884:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829770.434887:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790782080:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829770.434891:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790782208:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829770.434910:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829770.434918:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829770/real 1787829770] req@ffff9e6831aaad00 x1874668790782208/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829770.434926:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790782208/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829770.434933:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829770.434935:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829770.434937:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790782208:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829770.434941:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790782336:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829770.434957:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829770.434967:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829770/real 1787829770] req@ffff9e6831aaa1c0 x1874668790782336/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829770.434979:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790782336/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829770.434988:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829770.434991:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829770.434995:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790782336:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829770.435002:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790782464:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829770.435024:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829770.435036:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829770/real 1787829770] req@ffff9e6831aa8780 x1874668790782464/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829770.435049:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790782464/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829770.435057:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829770.435060:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829770.435064:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790782464:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829770.435085:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790782592:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829770.435103:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829770.435111:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829770/real 1787829770] req@ffff9e6831aaa940 x1874668790782592/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829770.435124:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790782592/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829770.435130:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829770.435133:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829770.435135:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790782592:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829770.945832:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829771.137854:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829771.969828:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829772.161890:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829772.993800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829773.185927:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829774.017796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829774.209904:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829775.041762:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829775.233857:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829775.553777:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829775.553787:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829775.553792:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829775.553800:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6412 00000100:00080000:2.0:1787829775.553808:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829775.553814:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829775.553845:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790782720/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829775.553872:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829775.553876:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829775.553878:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829775.553882:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6412 00000100:00080000:2.0:1787829775.553885:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829775.553889:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829775.553898:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829775.553910:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790782848/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829775.553914:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790782720:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829775.553928:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829775.553931:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829775.553932:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829775.553937:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6412 00000100:00080000:2.0:1787829775.553940:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829775.553944:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829775.553948:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829775.553957:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790782976/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829775.553967:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829775.553969:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829775.553973:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829775.553976:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829775.553981:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6412 00000100:00080000:2.0:1787829775.553984:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829775.553988:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829775.553992:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829775.554004:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790783104/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829775.554011:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829775/real 1787829775] req@ffff9e6804421e00 x1874668790782720/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829775.554017:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829775.554024:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829775.554026:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829775.554030:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790782720/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829775.554031:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6412 00000100:00080000:2.0:1787829775.554037:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829775.554042:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829775.554043:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787829775.554048:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:2.0:1787829775.554050:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829775.554052:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790782720:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829775.554065:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790783232/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829775.554074:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829775.554077:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829775.554079:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829775.554083:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6412 00000100:00080000:2.0:1787829775.554085:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829775.554089:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829775.554094:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790782848:192.168.203.109@tcp:38:kworker.0 00000008:00100000:2.0:1787829775.554094:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829775.554109:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790783360/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829775.554118:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829775.554124:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829775.554128:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787829775.554131:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829775/real 1787829775] req@ffff9e68044212c0 x1874668790782848/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829775.554132:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829775.554141:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6412 00000100:00080000:2.0:1787829775.554147:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829775.554149:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790782848/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829775.554154:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787829775.554162:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787829775.554163:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829775.554167:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829775.554171:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790782848:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829775.554178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790782976:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829775.554178:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790783488/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829775.554193:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829775.554198:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829775.554198:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829775.554203:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829775.554209:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6412 00000100:00100000:0.0:1787829775.554210:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829775/real 1787829775] req@ffff9e68044230c0 x1874668790782976/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829775.554214:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829775.554222:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829775.554226:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790782976/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829775.554228:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787829775.554240:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787829775.554244:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790783616/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829775.554245:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829775.554251:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790782976:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829775.554262:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790783104:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787829775.554278:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829775.554282:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829775.554284:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829775.554289:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6412 00000800:00000100:0.0:1787829775.554292:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829775.554295:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829775.554301:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829775.554303:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829775/real 1787829775] req@ffff9e6804420b40 x1874668790783104/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829775.554308:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829775.554313:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790783104/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829775.554320:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829775.554324:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829775.554325:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790783744/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829775.554327:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790783104:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829775.554334:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790783232:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829775.554345:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829775.554353:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829775/real 1787829775] req@ffff9e6804421680 x1874668790783232/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829775.554361:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790783232/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829775.554366:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829775.554368:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829775.554370:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790783232:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829775.554380:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790783360:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829775.554390:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829775.554395:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829775/real 1787829775] req@ffff9e6804422580 x1874668790783360/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829775.554401:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790783360/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829775.554409:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829775.554412:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829775.554415:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790783360:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829775.554420:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790783488:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829775.554438:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829775.554448:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829775/real 1787829775] req@ffff9e6804420f00 x1874668790783488/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829775.554457:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790783488/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829775.554462:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829775.554463:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829775.554465:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790783488:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829775.554468:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790783616:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829775.554480:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829775.554486:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829775/real 1787829775] req@ffff9e6804423480 x1874668790783616/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829775.554492:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790783616/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829775.554497:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829775.554498:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829775.554500:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790783616:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829775.554502:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790783744:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829775.554511:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829775.554516:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829775/real 1787829775] req@ffff9e6804420780 x1874668790783744/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829775.554521:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790783744/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829775.554526:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829775.554527:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829775.554528:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790783744:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829776.065830:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829776.257903:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829777.089813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829777.281846:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829778.113793:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829778.305880:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829779.137830:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829779.329820:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829780.161837:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829780.353918:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829780.673856:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829780.673869:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829780.673874:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829780.673886:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6417 00000100:00080000:3.0:1787829780.673899:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829780.673907:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829780.673952:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790783872/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829780.673985:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829780.673989:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829780.673991:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829780.673995:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6417 00000100:00080000:3.0:1787829780.673999:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829780.674004:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829780.674014:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829780.674028:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790784000/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829780.674044:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790783872:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829780.674049:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829780.674053:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829780.674055:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829780.674063:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6417 00000100:00080000:3.0:1787829780.674069:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829780.674073:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829780.674078:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829780.674091:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790784128/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829780.674100:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829780.674104:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829780.674106:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829780.674110:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6417 00000100:00080000:3.0:1787829780.674114:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829780.674118:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829780.674123:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829780.674134:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790784256/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829780.674140:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829780.674150:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829780.674154:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829780.674156:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829780.674162:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6417 00000100:00080000:3.0:1787829780.674167:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829780.674171:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829780.674176:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829780.674190:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790784384/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829780.674195:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829780/real 1787829780] req@ffff9e680a2d5a40 x1874668790783872/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829780.674203:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829780.674206:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829780.674208:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829780.674213:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6417 00000100:00100000:0.0:1787829780.674247:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790783872/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829780.674259:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829780.674264:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829780.674269:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790783872:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787829780.674297:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790784000:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829780.674321:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829780.674325:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829780.674336:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829780/real 1787829780] req@ffff9e680a2d6d00 x1874668790784000/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829780.674347:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790784000/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829780.674356:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829780.674360:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829780.674368:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790784000:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829780.674368:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829780.674378:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829780.674380:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790784128:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829780.674398:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790784512/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829780.674409:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829780.674411:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829780.674416:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829780.674418:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829780.674425:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829780/real 1787829780] req@ffff9e680a2d4f00 x1874668790784128/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829780.674425:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6417 00000100:00080000:3.0:1787829780.674431:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829780.674438:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829780.674445:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829780.674449:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790784128/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829780.674459:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790784640/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829780.674461:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829780.674470:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829780.674477:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790784128:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829780.674489:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790784256:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829780.674493:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829780.674498:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829780.674500:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829780.674505:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6417 00000100:00080000:3.0:1787829780.674510:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829780.674514:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787829780.674519:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787829780.674522:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829780.674535:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829780/real 1787829780] req@ffff9e680a2d52c0 x1874668790784256/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829780.674538:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790784768/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829780.674561:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829780.674565:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790784256/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829780.674567:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829780.674571:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829780.674580:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6417 00000100:00080000:0.0:1787829780.674584:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829780.674586:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829780.674592:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829780.674595:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829780.674600:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790784256:192.168.203.109@tcp:8:kworker.0 00000008:00100000:3.0:1787829780.674604:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829780.674610:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790784384:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829780.674619:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790784896/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829780.674636:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829780.674650:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829780/real 1787829780] req@ffff9e680a2d61c0 x1874668790784384/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829780.674663:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790784384/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829780.674670:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829780.674673:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829780.674677:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790784384:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829780.674726:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790784512:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829780.674749:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829780.674762:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829780/real 1787829780] req@ffff9e680a2d5a40 x1874668790784512/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829780.674775:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790784512/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829780.674784:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829780.674787:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829780.674790:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790784512:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829780.674796:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790784640:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829780.674814:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829780.674823:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829780/real 1787829780] req@ffff9e680a2d5e00 x1874668790784640/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829780.674834:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790784640/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829780.674842:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829780.674854:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829780.674857:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790784640:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829780.674862:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790784768:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829780.674878:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829780.674886:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829780/real 1787829780] req@ffff9e680a2d7480 x1874668790784768/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829780.674896:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790784768/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829780.674910:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829780.674913:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829780.674916:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790784768:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829780.674921:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790784896:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829780.674935:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829780.674944:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829780/real 1787829780] req@ffff9e680a2d4b40 x1874668790784896/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829780.674953:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790784896/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829780.674961:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829780.674965:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829780.674967:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790784896:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829781.185814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829781.377881:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829782.209806:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829782.401880:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829783.233829:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829783.425905:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829784.257810:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829784.449874:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829785.281802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829785.473850:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829785.793776:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829785.793785:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829785.793789:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829785.793803:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6422 00000100:00080000:3.0:1787829785.793814:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829785.793821:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829785.793849:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790785024/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829785.793868:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829785.793871:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829785.793872:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829785.793874:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6422 00000100:00080000:3.0:1787829785.793876:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829785.793878:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829785.793886:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829785.793894:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790785152/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829785.793902:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829785.793904:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829785.793905:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829785.793911:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6422 00000100:00080000:3.0:1787829785.793917:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829785.793918:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790785024:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829785.793921:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829785.793930:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829785.793940:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790785280/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829785.793946:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829785.793948:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829785.793949:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829785.793954:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6422 00000100:00080000:3.0:1787829785.793956:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829785.793958:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829785.793961:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829785.793970:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790785408/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829785.793976:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829785.793977:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829785.793978:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829785.793981:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6422 00000100:00080000:3.0:1787829785.793983:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829785.793985:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829785.793988:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829785.794002:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790785536/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829785.794007:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829785.794015:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829785.794016:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829785.794024:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829785.794028:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6422 00000100:00080000:3.0:1787829785.794032:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829785.794035:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829785.794040:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829785.794049:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790785664/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829785.794054:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829785.794056:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829785.794057:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829785.794061:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6422 00000100:00080000:3.0:1787829785.794062:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829785.794065:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829785.794068:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829785.794075:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790785792/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829785.794083:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829785/real 1787829785] req@ffff9e680a2d70c0 x1874668790785024/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829785.794087:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829785.794091:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829785.794094:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829785.794102:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6422 00000100:00080000:3.0:1787829785.794109:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829785.794120:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829785.794125:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829785.794139:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790785024/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829785.794139:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790785920/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829785.794186:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829785.794189:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829785.794203:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829785.794205:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829785.794210:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6422 00000100:00080000:3.0:1787829785.794216:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829785.794218:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829785.794223:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829785.794231:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790785024:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787829785.794231:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829785.794250:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790786048/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829785.794273:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790785152:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829785.794299:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829785.794312:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829785/real 1787829785] req@ffff9e680a2d6580 x1874668790785152/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829785.794331:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790785152/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829785.794341:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829785.794344:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829785.794348:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790785152:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829785.794354:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790785280:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829785.794371:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829785.794381:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829785/real 1787829785] req@ffff9e680a2d4000 x1874668790785280/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829785.794392:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790785280/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829785.794400:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829785.794403:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829785.794406:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790785280:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829785.794412:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790785408:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829785.794430:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829785.794440:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829785/real 1787829785] req@ffff9e680a2d43c0 x1874668790785408/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829785.794452:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790785408/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829785.794460:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829785.794463:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829785.794466:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790785408:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829785.794472:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790785536:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829785.794489:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829785.794498:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829785/real 1787829785] req@ffff9e680a2d4780 x1874668790785536/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829785.794510:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790785536/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829785.794518:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829785.794521:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829785.794524:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790785536:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829785.794530:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790785664:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829785.794548:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829785.794559:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829785/real 1787829785] req@ffff9e680a2d7c00 x1874668790785664/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829785.794570:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790785664/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829785.794586:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829785.794589:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829785.794593:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790785664:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829785.794599:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790785792:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829785.794613:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829785.794622:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829785/real 1787829785] req@ffff9e680a2d4b40 x1874668790785792/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829785.794632:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790785792/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829785.794640:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829785.794643:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829785.794646:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790785792:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829785.794652:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790785920:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829785.794669:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829785.794680:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829785/real 1787829785] req@ffff9e680a2d7480 x1874668790785920/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829785.794691:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790785920/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829785.794719:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829785.794722:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829785.794725:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790785920:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829785.794731:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790786048:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829785.794748:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829785.794758:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829785/real 1787829785] req@ffff9e680a2d5e00 x1874668790786048/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829785.794768:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790786048/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829785.794776:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829785.794779:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829785.794782:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790786048:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829786.305792:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829786.497859:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829787.329766:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829787.521956:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829788.353814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829788.546221:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829789.377810:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829789.569917:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829790.401843:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829790.593890:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829790.913868:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829790.913883:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829790.913888:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829790.913902:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6427 00000100:00080000:3.0:1787829790.913916:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829790.913926:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829790.913975:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790786176/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829790.914007:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829790.914012:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829790.914014:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829790.914018:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6427 00000100:00080000:3.0:1787829790.914023:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829790.914029:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829790.914041:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829790.914060:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790786304/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829790.914063:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790786176:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829790.914091:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829790.914095:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829790.914099:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829790.914106:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6427 00000100:00080000:3.0:1787829790.914112:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829790.914118:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829790.914126:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829790.914141:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790786432/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829790.914167:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829790.914181:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829790.914185:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829790.914189:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829790.914197:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6427 00000100:00080000:3.0:1787829790.914202:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829790.914208:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829790.914214:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829790/real 1787829790] req@ffff9e680a2d5a40 x1874668790786176/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829790.914216:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829790.914232:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790786176/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829790.914243:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829790.914247:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829790.914252:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790786560/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829790.914257:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790786176:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829790.914279:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829790.914283:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829790.914286:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829790.914291:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790786304:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829790.914296:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6427 00000100:00080000:3.0:1787829790.914304:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829790.914312:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:0.0:1787829790.914319:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787829790.914319:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829790.914332:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829790/real 1787829790] req@ffff9e680a2d61c0 x1874668790786304/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829790.914342:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790786688/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829790.914345:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790786304/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829790.914393:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829790.914397:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829790.914399:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829790.914405:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829790.914408:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829790.914415:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6427 00000100:00100000:0.0:1787829790.914420:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790786304:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829790.914424:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829790.914430:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790786432:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829790.914435:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829790.914445:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829790.914456:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829790.914464:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790786816/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829790.914470:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829790/real 1787829790] req@ffff9e680a2d52c0 x1874668790786432/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829790.914510:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829790.914515:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790786432/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829790.914521:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829790.914524:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829790.914529:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6427 00000100:00080000:3.0:1787829790.914538:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829790.914546:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829790.914549:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787829790.914554:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829790.914558:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790786432:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787829790.914560:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829790.914565:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790786560:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829790.914583:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790786944/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829790.914588:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829790.914600:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829790/real 1787829790] req@ffff9e680a2d4f00 x1874668790786560/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829790.914666:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829790.914669:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790786560/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829790.914674:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829790.914679:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829790.914691:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829790.914721:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829790.914725:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790786560:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829790.914731:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6427 00000100:00080000:3.0:1787829790.914738:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829790.914750:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790786688:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829790.914750:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829790.914771:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829790.914784:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829790.914796:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790787072/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829790.914798:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829790/real 1787829790] req@ffff9e680a2d6d00 x1874668790786688/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829790.914853:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829790.914858:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790786688/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829790.914864:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829790.914873:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829790.914877:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829790.914885:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6427 00000100:00080000:0.0:1787829790.914891:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829790.914897:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790786688:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829790.914898:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829790.914905:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790786816:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829790.914909:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829790.914920:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829790.914927:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829790.914940:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790787200/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829790.914941:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829790/real 1787829790] req@ffff9e680a2d5a40 x1874668790786816/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829790.915022:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790786816/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829790.915032:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829790.915035:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829790.915037:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790786816:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829790.915045:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790786944:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829790.915058:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829790.915066:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829790/real 1787829790] req@ffff9e680a2d5e00 x1874668790786944/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829790.915074:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790786944/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829790.915079:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829790.915081:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829790.915082:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790786944:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829790.915092:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790787072:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829790.915102:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829790.915108:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829790/real 1787829790] req@ffff9e680a2d7480 x1874668790787072/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829790.915115:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790787072/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829790.915119:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829790.915121:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829790.915123:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790787072:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829790.915127:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790787200:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829790.915136:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829790.915142:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829790/real 1787829790] req@ffff9e680a2d4b40 x1874668790787200/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829790.915150:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790787200/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829790.915155:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829790.915173:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829790.915175:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790787200:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829791.425847:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829791.617846:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829792.449795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829792.641846:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829793.473794:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829793.665870:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829794.497794:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829794.689853:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829795.521799:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829795.713833:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829796.033810:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829796.033819:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829796.033823:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829796.033839:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6432 00000100:00080000:1.0:1787829796.033855:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829796.033863:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829796.033900:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790787328/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829796.033922:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829796.033925:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829796.033927:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829796.033930:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6432 00000100:00080000:1.0:1787829796.033933:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829796.033936:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829796.033944:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829796.033955:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790787456/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829796.033962:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829796.033965:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829796.033966:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829796.033969:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6432 00000100:00080000:1.0:1787829796.033970:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829796.033975:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829796.033977:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790787328:192.168.203.109@tcp:250:kworker.0 00000008:00100000:1.0:1787829796.033979:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829796.033989:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790787584/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829796.033996:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829796.033999:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829796.034000:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829796.034003:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6432 00000100:00080000:1.0:1787829796.034005:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829796.034007:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829796.034011:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829796.034019:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790787712/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829796.034025:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829796.034028:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829796.034029:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829796.034032:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6432 00000100:00080000:1.0:1787829796.034034:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829796.034036:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829796.034039:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787829796.034044:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829796.034048:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790787840/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829796.034055:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829796.034058:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829796.034060:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829796.034062:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6432 00000100:00080000:1.0:1787829796.034067:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829796.034069:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829796.034072:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829796.034079:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790787968/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829796.034085:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829796.034087:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829796.034089:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829796.034092:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829796/real 1787829796] req@ffff9e6831aa9680 x1874668790787328/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829796.034093:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6432 00000100:00080000:1.0:1787829796.034098:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829796.034102:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829796.034108:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829796.034136:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790788096/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829796.034142:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790787328/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829796.034146:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829796.034154:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829796.034158:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829796.034165:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6432 00000100:00080000:0.0:1787829796.034168:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829796.034169:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829796.034174:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787829796.034178:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:1.0:1787829796.034181:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829796.034186:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790787328:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787829796.034194:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790788224/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829796.034200:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790787456:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829796.034204:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829796.034208:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829796.034209:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829796.034212:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6432 00000100:00080000:1.0:1787829796.034215:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829796.034217:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829796.034221:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829796.034229:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790788352/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829796.034237:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829796.034253:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829796/real 1787829796] req@ffff9e6831aa8000 x1874668790787456/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829796.034267:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790787456/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829796.034282:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829796.034286:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829796.034289:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790787456:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829796.034312:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790787584:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829796.034331:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829796.034342:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829796/real 1787829796] req@ffff9e6831aa9a40 x1874668790787584/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829796.034354:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790787584/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829796.034362:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829796.034365:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829796.034368:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790787584:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829796.034374:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790787712:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829796.034401:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829796.034412:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829796/real 1787829796] req@ffff9e6831aabc00 x1874668790787712/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829796.034424:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790787712/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829796.034432:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829796.034435:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829796.034439:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790787712:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829796.034445:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790787840:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829796.034461:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829796.034471:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829796/real 1787829796] req@ffff9e6831aaa940 x1874668790787840/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829796.034483:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790787840/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829796.034492:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829796.034495:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829796.034499:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790787840:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829796.034505:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790787968:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829796.034532:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829796.034542:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829796/real 1787829796] req@ffff9e6831aa8780 x1874668790787968/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829796.034553:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790787968/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829796.034563:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829796.034566:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829796.034569:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790787968:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829796.034575:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790788096:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787829796.034593:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829796.034604:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829796/real 1787829796] req@ffff9e6831aaa1c0 x1874668790788096/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829796.034616:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790788096/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829796.034624:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829796.034627:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829796.034631:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790788096:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829796.034636:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790788224:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829796.034662:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829796.034673:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829796/real 1787829796] req@ffff9e6831aaad00 x1874668790788224/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829796.034684:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790788224/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829796.034692:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829796.034717:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829796.034721:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790788224:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829796.034728:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790788352:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787829796.034758:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829796.034770:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829796/real 1787829796] req@ffff9e6831aaa580 x1874668790788352/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829796.034783:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790788352/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829796.034792:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829796.034795:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829796.034798:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790788352:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829796.545774:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829796.737901:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829797.569802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829797.761873:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829798.593790:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829798.785915:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829799.617838:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829799.809880:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829800.641826:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829800.833901:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829801.153824:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829801.153832:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829801.153836:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829801.153844:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6437 00000100:00080000:0.0:1787829801.153853:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829801.153859:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829801.153892:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790788480/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829801.153913:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829801.153916:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829801.153917:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829801.153920:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6437 00000100:00080000:0.0:1787829801.153923:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829801.153926:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829801.153935:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829801.153945:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790788608/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829801.153954:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829801.153956:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829801.153958:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829801.153961:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6437 00000100:00080000:0.0:1787829801.153963:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829801.153969:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829801.153975:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790788480:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1787829801.153976:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829801.153988:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790788736/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829801.153995:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829801.153997:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829801.153998:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829801.154003:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6437 00000100:00080000:0.0:1787829801.154005:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829801.154007:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829801.154010:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829801.154018:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790788864/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829801.154024:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829801.154026:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829801.154028:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829801.154031:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6437 00000100:00080000:0.0:1787829801.154051:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829801.154053:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829801.154057:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829801.154063:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829801.154069:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790788992/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829801.154078:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829801.154081:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829801.154082:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829801.154097:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6437 00000100:00080000:0.0:1787829801.154101:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829801.154105:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829801.154110:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829801.154121:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790789120/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829801.154128:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829801/real 1787829801] req@ffff9e6803e98f00 x1874668790788480/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829801.154130:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829801.154142:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829801.154148:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829801.154166:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6437 00000100:00080000:0.0:1787829801.154178:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829801.154182:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829801.154187:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829801.154188:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790788480/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829801.154199:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790789248/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829801.154206:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829801.154207:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829801.154217:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829801.154219:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829801.154222:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829801.154228:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790788480:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829801.154238:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6437 00000100:00080000:0.0:1787829801.154245:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829801.154250:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829801.154256:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829801.154261:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790788608:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829801.154275:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790789376/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829801.154289:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829801.154294:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829801.154298:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829801.154301:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829801.154307:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829801/real 1787829801] req@ffff9e6803e9ad00 x1874668790788608/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829801.154312:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6437 00000100:00080000:0.0:1787829801.154323:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829801.154338:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829801.154338:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790788608/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829801.154354:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829801.154357:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829801.154361:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829801.154366:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790788608:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829801.154369:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790789504/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829801.154387:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790788736:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829801.154418:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829801.154431:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829801/real 1787829801] req@ffff9e6803e9bc00 x1874668790788736/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829801.154445:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790788736/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829801.154454:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829801.154457:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829801.154459:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790788736:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829801.154465:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790788864:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829801.154482:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829801.154490:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829801/real 1787829801] req@ffff9e6803e9a940 x1874668790788864/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829801.154498:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790788864/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829801.154504:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829801.154506:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829801.154508:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790788864:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829801.154512:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790788992:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829801.154523:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829801.154529:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829801/real 1787829801] req@ffff9e6803e992c0 x1874668790788992/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829801.154536:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790788992/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829801.154542:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829801.154544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829801.154546:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790788992:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829801.154549:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790789120:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829801.154560:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829801.154566:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829801/real 1787829801] req@ffff9e6803e98b40 x1874668790789120/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829801.154573:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790789120/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829801.154580:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829801.154583:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829801.154585:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790789120:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829801.154588:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790789248:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829801.154598:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829801.154604:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829801/real 1787829801] req@ffff9e6803e98780 x1874668790789248/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829801.154611:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790789248/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829801.154617:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829801.154618:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829801.154620:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790789248:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829801.154635:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790789376:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829801.154655:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829801.154662:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829801/real 1787829801] req@ffff9e6803e9a1c0 x1874668790789376/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829801.154672:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790789376/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829801.154681:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829801.154684:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829801.154688:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790789376:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829801.154721:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790789504:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829801.154740:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829801.154748:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829801/real 1787829801] req@ffff9e6803e9b840 x1874668790789504/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829801.154757:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790789504/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829801.154766:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829801.154770:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829801.154773:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790789504:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829801.665782:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829801.857898:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829802.689802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829802.881878:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829803.713793:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829803.905941:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829804.737821:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829804.929884:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829805.761794:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829805.953881:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829806.273821:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829806.273829:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829806.273833:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829806.273842:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6442 00000100:00080000:0.0:1787829806.273852:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829806.273858:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829806.273887:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790789632/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829806.273907:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829806.273911:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829806.273912:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829806.273916:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6442 00000100:00080000:0.0:1787829806.273919:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829806.273923:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829806.273932:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829806.273942:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790789632:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787829806.273946:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790789760/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829806.273956:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829806.273959:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829806.273961:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829806.273965:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6442 00000100:00080000:0.0:1787829806.273968:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829806.273971:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829806.273975:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829806.273983:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790789888/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829806.273991:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829806.273993:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829806.273994:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829806.273997:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6442 00000100:00080000:0.0:1787829806.274000:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829806.274004:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829806.274007:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829806.274012:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829806.274020:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790790016/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829806.274026:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829806.274029:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829806.274030:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829806.274033:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6442 00000100:00080000:0.0:1787829806.274035:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829806.274038:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829806.274042:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829806/real 1787829806] req@ffff9e6803e9b480 x1874668790789632/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829806.274043:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829806.274072:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790790144/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829806.274080:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829806.274081:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790789632/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829806.274084:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829806.274087:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829806.274093:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6442 00000100:00080000:0.0:1787829806.274098:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829806.274102:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787829806.274104:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787829806.274106:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829806.274112:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829806.274117:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790790272/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829806.274122:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790789632:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829806.274128:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829806.274131:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829806.274134:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829806.274138:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6442 00000100:00080000:0.0:1787829806.274141:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829806.274144:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829806.274146:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829806.274152:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790790400/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829806.274158:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790789760:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829806.274162:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829806.274166:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829806.274167:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829806.274170:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6442 00000100:00080000:0.0:1787829806.274173:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829806.274175:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829806.274178:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829806.274186:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790790528/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829806.274194:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829806.274195:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829806.274200:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829806.274201:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829806.274205:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6442 00000100:00080000:0.0:1787829806.274210:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829806.274212:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829806.274216:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829806.274220:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829806/real 1787829806] req@ffff9e6803e99e00 x1874668790789760/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829806.274227:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790790656/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829806.274255:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790789760/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829806.274264:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829806.274268:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829806.274272:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790789760:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829806.274278:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790789888:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829806.274302:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829806.274317:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829806/real 1787829806] req@ffff9e6803e99680 x1874668790789888/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829806.274326:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790789888/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829806.274331:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829806.274335:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829806.274338:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790789888:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829806.274345:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790790016:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829806.274365:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829806.274376:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829806/real 1787829806] req@ffff9e6803e9a580 x1874668790790016/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829806.274388:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790790016/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829806.274397:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829806.274400:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829806.274402:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790790016:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829806.274406:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790790144:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829806.274420:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829806.274427:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829806/real 1787829806] req@ffff9e6803e9b840 x1874668790790144/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829806.274434:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790790144/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829806.274439:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829806.274441:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829806.274443:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790790144:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829806.274446:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790790272:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829806.274456:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829806.274461:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829806/real 1787829806] req@ffff9e6803e9a1c0 x1874668790790272/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829806.274468:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790790272/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829806.274478:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829806.274480:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829806.274482:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790790272:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829806.274494:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790790400:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829806.274504:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829806.274510:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829806/real 1787829806] req@ffff9e6803e98780 x1874668790790400/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829806.274517:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790790400/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829806.274522:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829806.274524:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829806.274525:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790790400:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829806.274529:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790790528:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829806.274538:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829806.274544:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829806/real 1787829806] req@ffff9e6803e98b40 x1874668790790528/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829806.274551:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790790528/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829806.274556:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829806.274557:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829806.274560:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790790528:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829806.274563:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790790656:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829806.274573:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829806.274578:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829806/real 1787829806] req@ffff9e6803e992c0 x1874668790790656/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829806.274584:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790790656/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829806.274589:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829806.274591:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829806.274593:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790790656:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829806.785815:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829806.977867:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829807.809823:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829808.001828:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829808.833805:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829809.025875:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829809.857781:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829810.049865:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829810.881827:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829811.073904:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829811.393860:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829811.393875:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829811.393880:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829811.393894:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6448 00000100:00080000:3.0:1787829811.393908:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829811.393918:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829811.393968:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790790784/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829811.394007:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829811.394012:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787829811.394078:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790790784:192.168.203.109@tcp:250:kworker.0 00000800:00000100:1.0:1787829811.394140:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829811.394176:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829811/real 1787829811] req@ffff9e680a2d7c00 x1874668790790784/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829811.394190:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790790784/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829811.394196:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829811.394209:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6448 00000100:00080000:1.0:1787829811.394212:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829811.394216:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829811.394223:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829811.394228:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829811.394237:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790790784:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787829811.394248:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829811.394269:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790790912/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829811.394286:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829811.394292:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829811.394294:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829811.394300:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6448 00000100:00100000:1.0:1787829811.394307:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790790912:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829811.394308:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829811.394319:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829811.394327:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829811.394347:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790791040/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829811.394350:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829811.394372:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829811/real 1787829811] req@ffff9e680a2d4780 x1874668790790912/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829811.394381:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829811.394390:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829811.394393:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829811.394399:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790790912/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829811.394403:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6448 00000100:00080000:3.0:1787829811.394414:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829811.394424:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787829811.394426:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787829811.394434:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829811.394435:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829811.394459:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790791168/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829811.394462:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790790912:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829811.394486:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829811.394491:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829811.394495:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829811.394498:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790791040:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829811.394504:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6448 00000100:00080000:3.0:1787829811.394513:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829811.394519:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829811.394526:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829811.394538:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829811.394558:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790791296/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829811.394559:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829811/real 1787829811] req@ffff9e680a2d43c0 x1874668790791040/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829811.394611:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829811.394618:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790791040/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829811.394619:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829811.394626:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829811.394646:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829811.394652:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829811.394655:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6448 00000100:00100000:1.0:1787829811.394660:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790791040:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829811.394664:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829811.394672:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790791168:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829811.394675:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829811.394689:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829811.394744:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829811.394768:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790791424/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829811.394772:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829811/real 1787829811] req@ffff9e680a2d4000 x1874668790791168/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829811.394807:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829811.394813:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787829811.394816:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790791168/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829811.394817:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829811.394832:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6448 00000100:00080000:3.0:1787829811.394841:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829811.394845:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829811.394851:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787829811.394857:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:3.0:1787829811.394862:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829811.394866:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790791168:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829811.394885:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790791552/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829811.394897:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790791296:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829811.394906:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829811.394912:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829811.394915:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829811.394922:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6448 00000100:00080000:3.0:1787829811.394929:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829811.394935:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829811.394940:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829811.394949:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829811.394959:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829811/real 1787829811] req@ffff9e680a2d6580 x1874668790791296/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829811.394968:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790791680/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829811.394983:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790791296/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829811.394988:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829811.395001:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829811.395006:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829811.395010:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829811.395031:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6448 00000100:00080000:3.0:1787829811.395042:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829811.395043:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829811.395055:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829811.395056:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790791296:192.168.203.109@tcp:8:kworker.0 00000008:00100000:3.0:1787829811.395072:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829811.395078:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790791424:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829811.395094:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790791808/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829811.395112:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829811.395132:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829811/real 1787829811] req@ffff9e680a2d70c0 x1874668790791424/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829811.395146:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790791424/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829811.395164:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829811.395168:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829811.395172:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790791424:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829811.395190:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790791552:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829811.395214:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829811.395227:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829811/real 1787829811] req@ffff9e680a2d4780 x1874668790791552/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829811.395240:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790791552/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829811.395249:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829811.395252:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829811.395256:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790791552:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829811.395263:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790791680:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829811.395287:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829811.395300:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829811/real 1787829811] req@ffff9e680a2d7c00 x1874668790791680/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829811.395313:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790791680/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829811.395322:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829811.395325:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829811.395329:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790791680:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829811.395336:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790791808:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829811.395357:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829811.395369:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829811/real 1787829811] req@ffff9e680a2d4b40 x1874668790791808/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829811.395381:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790791808/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829811.395390:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829811.395393:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829811.395396:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790791808:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829811.905783:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829812.097911:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829812.929850:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829813.121903:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829813.953765:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829814.145934:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829814.977861:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829815.169823:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829816.001814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829816.193840:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829816.513836:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829816.513847:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829816.513852:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829816.513863:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6453 00000100:00080000:3.0:1787829816.513873:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829816.513879:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829816.513914:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790791936/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829816.513937:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829816.513940:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829816.513942:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829816.513945:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6453 00000100:00080000:3.0:1787829816.513947:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829816.513950:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829816.513958:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829816.513965:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790792064/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829816.513972:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829816.513974:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829816.513975:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829816.513997:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6453 00000100:00080000:3.0:1787829816.514000:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829816.514003:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829816.514007:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829816.514015:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790792192/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829816.514024:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829816.514027:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790791936:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829816.514027:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829816.514030:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829816.514037:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6453 00000100:00080000:3.0:1787829816.514040:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829816.514043:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829816.514046:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829816.514056:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790792320/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829816.514062:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829816.514064:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829816.514065:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829816.514068:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6453 00000100:00080000:3.0:1787829816.514070:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829816.514072:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829816.514075:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829816.514081:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790792448/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829816.514087:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829816.514089:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829816.514090:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829816.514092:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6453 00000100:00080000:3.0:1787829816.514096:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829816.514102:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829816.514102:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829816.514109:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829816.514117:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790792576/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829816.514124:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829816.514126:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829816.514128:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829816.514131:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6453 00000100:00080000:3.0:1787829816.514135:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829816.514139:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829816.514145:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829816.514151:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829816/real 1787829816] req@ffff9e680a2d7480 x1874668790791936/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829816.514159:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790792704/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829816.514180:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829816.514183:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829816.514186:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829816.514188:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790791936/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829816.514195:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6453 00000100:00080000:3.0:1787829816.514203:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829816.514210:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829816.514211:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787829816.514221:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:3.0:1787829816.514222:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829816.514231:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790791936:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829816.514242:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790792832/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829816.514249:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790792064:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829816.514262:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829816.514267:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829816.514269:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829816.514274:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6453 00000100:00080000:3.0:1787829816.514278:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829816.514281:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829816.514286:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829816.514287:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829816.514300:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790792960/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829816.514309:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829816/real 1787829816] req@ffff9e680a2d5e00 x1874668790792064/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829816.514327:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790792064/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829816.514337:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829816.514341:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829816.514345:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790792064:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829816.514371:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790792192:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829816.514394:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829816.514406:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829816/real 1787829816] req@ffff9e680a2d5a40 x1874668790792192/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829816.514418:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790792192/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829816.514426:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829816.514430:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829816.514433:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790792192:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829816.514440:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790792320:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829816.514462:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829816.514474:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829816/real 1787829816] req@ffff9e680a2d6d00 x1874668790792320/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829816.514487:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790792320/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829816.514496:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829816.514499:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829816.514502:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790792320:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829816.514509:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790792448:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829816.514530:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829816.514542:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829816/real 1787829816] req@ffff9e680a2d4f00 x1874668790792448/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829816.514554:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790792448/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829816.514563:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829816.514566:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829816.514570:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790792448:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829816.514576:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790792576:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829816.514597:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829816.514609:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829816/real 1787829816] req@ffff9e680a2d52c0 x1874668790792576/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829816.514621:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790792576/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829816.514637:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829816.514640:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829816.514643:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790792576:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829816.514649:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790792704:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829816.514670:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829816.514681:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829816/real 1787829816] req@ffff9e680a2d61c0 x1874668790792704/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829816.514694:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790792704/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829816.514740:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829816.514744:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829816.514748:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790792704:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829816.514754:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790792832:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829816.514776:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829816.514789:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829816/real 1787829816] req@ffff9e680a2d4b40 x1874668790792832/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829816.514800:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790792832/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829816.514809:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829816.514812:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829816.514816:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790792832:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829816.514823:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790792960:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829816.514842:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829816.514853:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829816/real 1787829816] req@ffff9e680a2d7c00 x1874668790792960/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829816.514865:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790792960/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829816.514873:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829816.514877:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829816.514880:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790792960:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829817.025813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829817.217870:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829818.049814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829818.241892:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829819.073845:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829819.265895:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829820.097809:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829820.289880:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829821.121787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829821.313920:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829821.633834:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829821.633842:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829821.633846:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829821.633854:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6458 00000100:00080000:2.0:1787829821.633863:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829821.633873:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829821.633905:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790793088/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829821.633931:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829821.633934:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829821.633935:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829821.633938:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6458 00000100:00080000:2.0:1787829821.633941:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829821.633955:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829821.633966:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829821.633982:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790793088:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829821.633984:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790793216/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829821.634000:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829821.634004:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829821.634007:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829821.634011:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6458 00000100:00080000:2.0:1787829821.634016:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829821.634020:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829821.634027:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829821.634038:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790793344/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829821.634048:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829821.634051:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829821.634056:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829821.634058:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829821.634064:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6458 00000100:00080000:2.0:1787829821.634068:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829821.634071:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829821.634076:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829821.634085:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829821/real 1787829821] req@ffff9e6804422940 x1874668790793088/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829821.634088:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790793472/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829821.634112:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829821.634115:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829821.634118:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790793088/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829821.634118:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829821.634124:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6458 00000100:00080000:2.0:1787829821.634130:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829821.634136:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787829821.634138:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787829821.634139:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829821.634144:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829821.634149:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790793088:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829821.634152:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790793600/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829821.634159:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829821.634162:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829821.634163:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829821.634165:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6458 00000100:00080000:2.0:1787829821.634168:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829821.634174:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829821.634175:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790793216:192.168.203.109@tcp:38:kworker.0 00000008:00100000:2.0:1787829821.634179:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829821.634190:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790793728/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829821.634199:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829821.634201:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829821.634203:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829821.634206:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829821.634212:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6458 00000100:00100000:1.0:1787829821.634217:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829821/real 1787829821] req@ffff9e68044221c0 x1874668790793216/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829821.634218:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829821.634228:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829821.634232:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790793216/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829821.634236:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829821.634248:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787829821.634251:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790793856/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829821.634254:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829821.634260:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790793216:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829821.634271:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790793344:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829821.634274:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829821.634277:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829821.634279:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829821.634285:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6458 00000100:00080000:2.0:1787829821.634289:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829821.634291:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829821.634295:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829821.634301:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829821.634304:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829821/real 1787829821] req@ffff9e6804423840 x1874668790793344/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829821.634314:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790793984/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829821.634316:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790793344/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829821.634339:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829821.634340:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829821.634343:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829821.634349:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829821.634350:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790793344:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829821.634355:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829821.634359:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790793472:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787829821.634364:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6458 00000100:00080000:2.0:1787829821.634370:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829821.634377:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829821.634378:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829821.634386:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829821.634388:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829821/real 1787829821] req@ffff9e6804422d00 x1874668790793472/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829821.634399:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790793472/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829821.634404:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790794112/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829821.634408:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829821.634415:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829821.634421:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790793472:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829821.634428:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790793600:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829821.634448:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829821.634455:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829821/real 1787829821] req@ffff9e6804420780 x1874668790793600/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829821.634462:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790793600/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829821.634468:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829821.634470:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829821.634471:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790793600:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829821.634482:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790793728:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829821.634493:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829821.634499:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829821/real 1787829821] req@ffff9e6804423480 x1874668790793728/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829821.634506:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790793728/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829821.634516:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829821.634518:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829821.634520:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790793728:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829821.634524:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790793856:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829821.634533:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829821.634539:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829821/real 1787829821] req@ffff9e6804420f00 x1874668790793856/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829821.634545:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790793856/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829821.634550:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829821.634552:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829821.634554:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790793856:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829821.634557:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790793984:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829821.634566:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829821.634571:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829821/real 1787829821] req@ffff9e6804422580 x1874668790793984/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829821.634578:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790793984/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829821.634583:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829821.634585:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829821.634587:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790793984:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829821.634591:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790794112:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829821.634601:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829821.634606:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829821/real 1787829821] req@ffff9e6804421680 x1874668790794112/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829821.634612:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790794112/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829821.634618:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829821.634619:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829821.634621:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790794112:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829822.145823:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829822.337827:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829823.169761:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829823.361883:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829824.193781:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829824.385890:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829825.217826:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829825.409898:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829826.241777:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829826.433888:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829826.753791:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829826.753805:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829826.753810:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829826.753824:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6463 00000100:00080000:3.0:1787829826.753837:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829826.753844:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829826.753891:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790794240/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829826.753926:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829826.753929:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829826.753931:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829826.753934:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6463 00000100:00080000:3.0:1787829826.753936:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829826.753939:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829826.753948:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829826.753961:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790794368/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829826.753968:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829826.753971:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829826.753972:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829826.753975:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6463 00000100:00080000:3.0:1787829826.753977:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829826.753978:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829826.753981:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829826.753986:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790794240:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829826.753989:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790794496/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829826.753999:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829826.754002:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829826.754003:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829826.754005:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6463 00000100:00080000:3.0:1787829826.754007:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829826.754009:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829826.754011:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829826.754020:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790794624/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829826.754024:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829826.754026:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829826.754028:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829826.754030:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6463 00000100:00080000:3.0:1787829826.754033:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829826.754035:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829826.754038:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829826.754044:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790794752/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829826.754049:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829826.754051:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829826.754052:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829826.754055:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6463 00000100:00080000:3.0:1787829826.754057:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829826.754060:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829826.754062:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829826.754070:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790794880/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829826.754071:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829826.754079:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829826.754082:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829826.754083:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829826.754086:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6463 00000100:00080000:3.0:1787829826.754088:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829826.754097:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829826.754100:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829826.754108:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790795008/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829826.754113:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829826.754116:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829826.754117:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829826.754119:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6463 00000100:00080000:3.0:1787829826.754122:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829826.754124:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829826/real 1787829826] req@ffff9e680a2d4780 x1874668790794240/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829826.754127:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829826.754133:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829826.754144:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790795136/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829826.754149:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790794240/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829826.754155:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829826.754163:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829826.754166:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829826.754170:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829826.754170:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6463 00000100:00080000:3.0:1787829826.754176:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829826.754179:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829826.754182:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829826.754186:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790794240:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787829826.754186:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829826.754197:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790795264/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829826.754199:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790794368:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829826.754228:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829826.754242:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829826/real 1787829826] req@ffff9e680a2d70c0 x1874668790794368/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829826.754255:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790794368/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829826.754272:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829826.754276:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829826.754279:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790794368:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829826.754307:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790794496:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829826.754337:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829826.754348:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829826/real 1787829826] req@ffff9e680a2d6580 x1874668790794496/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829826.754359:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790794496/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829826.754367:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829826.754370:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829826.754374:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790794496:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829826.754379:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790794624:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829826.754395:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829826.754403:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829826/real 1787829826] req@ffff9e680a2d4000 x1874668790794624/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829826.754415:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790794624/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829826.754429:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829826.754432:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829826.754436:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790794624:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829826.754441:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790794752:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829826.754461:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829826.754481:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829826/real 1787829826] req@ffff9e680a2d43c0 x1874668790794752/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829826.754496:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790794752/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829826.754503:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829826.754507:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829826.754510:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790794752:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829826.754515:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790794880:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829826.754536:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829826.754547:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829826/real 1787829826] req@ffff9e680a2d7c00 x1874668790794880/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829826.754558:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790794880/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829826.754573:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829826.754576:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829826.754580:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790794880:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829826.754585:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790795008:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829826.754613:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829826.754625:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829826/real 1787829826] req@ffff9e680a2d4b40 x1874668790795008/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829826.754635:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790795008/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829826.754643:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829826.754646:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829826.754649:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790795008:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829826.754654:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790795136:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829826.754675:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829826.754685:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829826/real 1787829826] req@ffff9e680a2d61c0 x1874668790795136/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829826.754723:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790795136/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829826.754732:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829826.754735:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829826.754739:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790795136:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829826.754744:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790795264:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829826.754763:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829826.754774:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829826/real 1787829826] req@ffff9e680a2d52c0 x1874668790795264/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829826.754785:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790795264/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829826.754792:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829826.754795:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829826.754798:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790795264:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829827.265826:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829827.457895:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829828.289795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829828.481909:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829829.313833:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829829.505855:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829830.337799:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829830.530007:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829831.361805:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829831.553897:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829831.873794:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829831.873804:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829831.873808:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829831.873816:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6468 00000100:00080000:3.0:1787829831.873827:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829831.873834:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829831.873889:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790795392/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829831.873918:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829831.873922:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829831.873924:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829831.873928:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6468 00000100:00080000:3.0:1787829831.873932:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829831.873935:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829831.873944:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829831.873958:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790795520/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829831.873961:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790795392:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829831.873977:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829831.873980:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829831.873982:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829831.873988:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6468 00000100:00080000:3.0:1787829831.873992:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829831.873996:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829831.874002:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829831.874014:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790795648/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829831.874024:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829831.874027:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829831.874029:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829831.874032:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6468 00000100:00080000:3.0:1787829831.874036:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829831.874040:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829831.874045:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829831.874057:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790795776/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829831.874059:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790795520:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829831.874075:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829831.874079:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829831.874081:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829831.874087:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6468 00000100:00080000:3.0:1787829831.874091:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829831.874095:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790795648:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829831.874098:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829831.874108:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829831.874122:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790795904/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829831.874128:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790795776:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829831.874137:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829831.874150:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829831.874153:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829831.874158:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6468 00000100:00080000:3.0:1787829831.874163:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829831.874165:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790795904:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829831.874170:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829831.874177:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829831.874192:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790796032/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829831.874211:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829831.874215:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829831.874216:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829831.874220:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6468 00000100:00080000:3.0:1787829831.874223:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829831.874227:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829831.874233:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829831.874238:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790796032:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829831.874246:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790796160/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829831.874258:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829831.874262:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787829831.874264:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790796160:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829831.874272:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829831.874279:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6468 00000100:00080000:3.0:1787829831.874283:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829831.874287:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829831.874292:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829831.874304:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790796288/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829831.874319:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829831.874322:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829831.874324:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829831.874329:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6468 00000100:00080000:3.0:1787829831.874332:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829831.874336:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829831.874341:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790796288:192.168.203.109@tcp:8:kworker.0 00000008:00100000:3.0:1787829831.874342:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829831.874358:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790796416/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829831.874389:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790796416:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829832.385807:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829832.577855:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829833.409802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829833.601915:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829834.433793:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829834.625897:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787829834.946114:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:1.0:1787829834.946156:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787829834.946173:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829834.946224:0:2388:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:1.0:1787829834.946236:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829834.946243:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829834.946249:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829834.946257:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829834.946263:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829834.946270:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829834.946277:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787829834.946896:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829831/real 1787829834] req@ffff9e680a2d7c00 x1874668790796416/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829834.946923:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790796416/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829834.946935:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829834.946940:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829834.946944:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790796416:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829834.946953:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829831/real 1787829834] req@ffff9e680a2d4b40 x1874668790796288/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829834.946964:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790796288/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829834.946972:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829834.946976:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829834.946980:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790796288:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829834.946987:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829831/real 1787829834] req@ffff9e680a2d61c0 x1874668790796160/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829834.947000:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790796160/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829834.947007:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829834.947010:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829834.947013:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790796160:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829834.947019:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829831/real 1787829834] req@ffff9e680a2d52c0 x1874668790796032/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829834.947029:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790796032/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829834.947037:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829834.947039:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829834.947043:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790796032:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829834.947049:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829831/real 1787829834] req@ffff9e680a2d7480 x1874668790795904/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829834.947059:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790795904/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829834.947065:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829834.947068:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829834.947071:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790795904:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829834.947077:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829831/real 1787829834] req@ffff9e680a2d5e00 x1874668790795776/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829834.947087:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790795776/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829834.947094:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829834.947097:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829834.947100:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790795776:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829834.947107:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829831/real 1787829834] req@ffff9e680a2d6d00 x1874668790795520/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829834.947117:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790795520/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829834.947124:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829834.947126:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829834.947129:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790795520:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829834.947135:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829831/real 1787829834] req@ffff9e680a2d5a40 x1874668790795648/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829834.947145:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790795648/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829834.947150:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829834.947153:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829834.947156:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790795648:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829834.947161:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829831/real 1787829834] req@ffff9e680a2d4f00 x1874668790795392/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829834.947171:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790795392/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829834.947177:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829834.947185:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829834.947188:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790795392:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829835.457781:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829835.649833:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829836.481802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829836.673860:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829836.993828:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829836.993851:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829836.993857:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829836.993870:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6473 00000100:00080000:3.0:1787829836.993884:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829836.993894:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829836.993937:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790796544/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829836.993982:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829836.993986:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829836.993988:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829836.993992:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6473 00000100:00080000:3.0:1787829836.993997:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829836.994001:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829836.994012:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829836.994025:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790796672/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829836.994035:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829836.994039:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829836.994040:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829836.994044:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6473 00000100:00080000:3.0:1787829836.994048:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829836.994052:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829836.994057:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829836.994067:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790796800/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829836.994076:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829836.994079:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829836.994081:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829836.994085:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6473 00000100:00080000:3.0:1787829836.994089:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829836.994093:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829836.994098:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829836.994108:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790796928/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829836.994116:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829836.994119:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829836.994121:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829836.994125:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6473 00000100:00080000:3.0:1787829836.994128:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829836.994132:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829836.994136:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829836.994154:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790797056/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829836.994163:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829836.994166:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829836.994168:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829836.994171:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6473 00000100:00080000:3.0:1787829836.994175:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829836.994181:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829836.994185:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829836.994198:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790797184/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829836.994212:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829836.994216:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829836.994218:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829836.994221:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6473 00000100:00080000:3.0:1787829836.994224:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829836.994227:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829836.994231:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829836.994239:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790797312/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829836.994247:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829836.994250:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829836.994252:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829836.994255:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6473 00000100:00080000:3.0:1787829836.994258:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829836.994262:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829836.994266:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829836.994275:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790797440/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829836.994283:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829836.994287:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829836.994289:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829836.994292:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6473 00000100:00080000:3.0:1787829836.994296:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829836.994300:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829836.994304:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829836.994312:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790797568/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829836.994446:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790796544:192.168.203.109@tcp:250:kworker.0 00000800:00000100:2.0:1787829836.994521:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829836.994561:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829836/real 1787829836] req@ffff9e680a2d43c0 x1874668790796544/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829836.994578:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790796544/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829836.994589:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829836.994595:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829836.994598:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790796544:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829836.994606:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790796672:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829836.994629:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829836.994641:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829836/real 1787829836] req@ffff9e680a2d4000 x1874668790796672/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829836.994652:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790796672/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829836.994660:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829836.994664:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829836.994667:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790796672:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829836.994674:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790796800:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829836.994727:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829836.994740:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829836/real 1787829836] req@ffff9e680a2d6580 x1874668790796800/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829836.994750:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790796800/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829836.994757:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829836.994761:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829836.994763:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790796800:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829836.994770:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790796928:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829836.994790:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829836.994800:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829836/real 1787829836] req@ffff9e680a2d70c0 x1874668790796928/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829836.994810:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790796928/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829836.994817:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829836.994820:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829836.994822:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790796928:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829836.994828:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790797056:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829836.994857:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829836.994867:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829836/real 1787829836] req@ffff9e680a2d4780 x1874668790797056/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829836.994877:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790797056/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829836.994884:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829836.994887:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829836.994890:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790797056:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829836.994896:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790797184:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829836.994914:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829836.994929:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829836/real 1787829836] req@ffff9e680a2d4f00 x1874668790797184/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829836.994944:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790797184/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829836.994953:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829836.994955:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829836.994958:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790797184:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829836.994963:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790797312:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829836.994980:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829836.994989:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829836/real 1787829836] req@ffff9e680a2d5a40 x1874668790797312/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829836.994999:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790797312/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829836.995006:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829836.995009:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829836.995011:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790797312:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829836.995017:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790797440:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829836.995034:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829836.995044:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829836/real 1787829836] req@ffff9e680a2d6d00 x1874668790797440/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829836.995054:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790797440/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829836.995065:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829836.995068:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829836.995071:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790797440:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829836.995077:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790797568:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829836.995094:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829836.995104:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829836/real 1787829836] req@ffff9e680a2d5e00 x1874668790797568/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829836.995115:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790797568/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829836.995123:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829836.995126:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829836.995129:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790797568:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829837.505844:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829837.697900:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829838.529801:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829838.721814:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829839.553802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829839.745841:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829840.577788:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829840.769851:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829841.601786:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829841.793937:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829842.113784:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829842.113791:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829842.113795:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829842.113816:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6478 00000100:00080000:3.0:1787829842.113825:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829842.113875:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829842.113904:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790797696/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829842.113924:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829842.113927:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829842.113929:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829842.113932:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6478 00000100:00080000:3.0:1787829842.113935:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829842.113938:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829842.113946:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829842.113956:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790797824/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829842.113962:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829842.113964:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829842.113965:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829842.113967:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6478 00000100:00080000:3.0:1787829842.113968:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829842.113970:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829842.113973:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829842.113980:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790797952/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829842.113986:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829842.113988:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829842.113989:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829842.113990:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6478 00000100:00080000:3.0:1787829842.113992:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829842.113994:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829842.113997:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829842.114004:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790798080/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829842.114010:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829842.114011:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829842.114013:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829842.114017:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6478 00000100:00080000:3.0:1787829842.114019:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829842.114022:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829842.114024:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829842.114031:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790798208/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829842.114036:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829842.114037:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829842.114039:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829842.114041:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6478 00000100:00080000:3.0:1787829842.114043:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829842.114046:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829842.114049:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829842.114055:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790798336/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829842.114060:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829842.114062:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829842.114063:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829842.114066:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6478 00000100:00080000:3.0:1787829842.114067:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829842.114070:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829842.114072:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829842.114078:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790798464/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829842.114083:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829842.114084:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829842.114085:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829842.114087:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6478 00000100:00080000:3.0:1787829842.114088:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829842.114090:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829842.114092:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829842.114099:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790798592/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829842.114106:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829842.114109:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829842.114110:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829842.114113:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6478 00000100:00080000:3.0:1787829842.114116:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829842.114121:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790797696:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829842.114124:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829842.114132:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829842.114182:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829842.114210:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790798720/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829842.114220:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829842/real 1787829842] req@ffff9e680a2d7480 x1874668790797696/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829842.114240:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790797696/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829842.114250:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829842.114255:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829842.114259:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790797696:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829842.114266:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790797824:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829842.114299:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829842.114308:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829842/real 1787829842] req@ffff9e680a2d52c0 x1874668790797824/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829842.114316:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790797824/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829842.114322:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829842.114324:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829842.114326:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790797824:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829842.114330:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790797952:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829842.114342:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829842.114349:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829842/real 1787829842] req@ffff9e680a2d61c0 x1874668790797952/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829842.114356:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790797952/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829842.114361:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829842.114363:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829842.114365:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790797952:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829842.114369:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790798080:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829842.114380:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829842.114387:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829842/real 1787829842] req@ffff9e680a2d4b40 x1874668790798080/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829842.114395:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790798080/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829842.114402:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829842.114404:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829842.114406:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790798080:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829842.114408:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790798208:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829842.114417:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829842.114422:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829842/real 1787829842] req@ffff9e680a2d7c00 x1874668790798208/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829842.114428:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790798208/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829842.114433:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829842.114436:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829842.114438:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790798208:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829842.114442:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790798336:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829842.114453:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829842.114459:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829842/real 1787829842] req@ffff9e680a2d5e00 x1874668790798336/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829842.114465:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790798336/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829842.114474:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829842.114476:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829842.114478:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790798336:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829842.114481:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790798464:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829842.114489:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829842.114494:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829842/real 1787829842] req@ffff9e680a2d6d00 x1874668790798464/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829842.114499:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790798464/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829842.114504:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829842.114505:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829842.114507:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790798464:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829842.114510:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790798592:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829842.114519:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829842.114527:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829842/real 1787829842] req@ffff9e680a2d5a40 x1874668790798592/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829842.114536:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790798592/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829842.114543:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829842.114546:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829842.114550:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790798592:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829842.114578:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790798720:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829842.114594:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829842.114604:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829842/real 1787829842] req@ffff9e680a2d4f00 x1874668790798720/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829842.114615:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790798720/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829842.114623:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829842.114687:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829842.114690:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790798720:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829842.625781:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829842.817879:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829843.649836:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829843.841921:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829844.673776:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829844.865905:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829845.697802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829845.889877:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829846.721817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829846.913847:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829847.233855:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829847.233863:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829847.233866:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829847.233874:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6483 00000100:00080000:0.0:1787829847.233882:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829847.233887:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829847.233918:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790798848/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829847.233937:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829847.233940:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829847.233941:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829847.233945:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6483 00000100:00080000:0.0:1787829847.233954:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829847.233961:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829847.233970:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829847.233971:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790798848:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787829847.233981:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790798976/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829847.233990:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829847.233993:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829847.233994:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829847.233997:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6483 00000100:00080000:0.0:1787829847.234000:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829847.234003:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829847.234006:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829847.234014:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790799104/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829847.234020:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829847.234022:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829847.234024:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829847.234026:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829847.234029:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6483 00000100:00080000:0.0:1787829847.234031:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829847.234034:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829847.234037:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829847.234045:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790799232/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829847.234053:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829847.234056:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829847.234056:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829847/real 1787829847] req@ffff9e6803e9a940 x1874668790798848/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829847.234057:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829847.234060:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6483 00000100:00080000:0.0:1787829847.234063:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829847.234066:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829847.234070:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829847.234072:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790798848/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829847.234078:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790799360/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829847.234080:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829847.234084:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829847.234086:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829847.234087:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790798848:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829847.234089:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829847.234091:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829847.234094:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6483 00000100:00080000:0.0:1787829847.234096:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829847.234099:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829847.234103:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829847.234107:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790798976:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829847.234110:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790799488/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829847.234117:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829847.234120:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829847.234121:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829847.234125:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6483 00000800:00000100:2.0:1787829847.234125:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829847.234128:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829847.234131:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829847.234135:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829847.234135:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829847/real 1787829847] req@ffff9e6803e9bc00 x1874668790798976/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829847.234143:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790799616/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829847.234145:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790798976/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829847.234151:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829847.234153:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829847.234154:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829847.234156:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829847.234156:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829847.234159:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6483 00000100:00100000:2.0:1787829847.234160:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790798976:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829847.234162:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829847.234164:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790799104:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829847.234165:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829847.234169:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829847.234176:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790799744/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829847.234179:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829847.234184:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829847.234187:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829847.234188:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829847/real 1787829847] req@ffff9e6803e9ad00 x1874668790799104/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829847.234189:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829847.234193:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6483 00000100:00080000:0.0:1787829847.234195:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829847.234198:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829847.234198:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790799104/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829847.234202:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829847.234206:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829847.234209:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829847.234210:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790799872/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829847.234212:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790799104:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829847.234218:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790799232:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829847.234233:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829847.234243:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829847/real 1787829847] req@ffff9e6803e98f00 x1874668790799232/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829847.234254:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790799232/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829847.234259:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829847.234262:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829847.234264:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790799232:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829847.234267:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790799360:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829847.234279:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829847.234285:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829847/real 1787829847] req@ffff9e6803e992c0 x1874668790799360/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829847.234292:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790799360/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829847.234296:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829847.234298:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829847.234299:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790799360:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829847.234309:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790799488:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829847.234319:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829847.234324:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829847/real 1787829847] req@ffff9e6803e98b40 x1874668790799488/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829847.234331:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790799488/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829847.234336:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829847.234337:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829847.234339:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790799488:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829847.234343:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790799616:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829847.234351:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829847.234355:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829847/real 1787829847] req@ffff9e6803e98780 x1874668790799616/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829847.234362:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790799616/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829847.234366:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829847.234368:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829847.234369:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790799616:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829847.234372:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790799744:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829847.234380:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829847.234384:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829847/real 1787829847] req@ffff9e6803e9a1c0 x1874668790799744/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829847.234391:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790799744/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829847.234395:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829847.234396:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829847.234398:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790799744:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829847.234400:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790799872:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829847.234408:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829847.234413:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829847/real 1787829847] req@ffff9e6803e9b840 x1874668790799872/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829847.234419:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790799872/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829847.234423:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829847.234425:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829847.234426:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790799872:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829847.745819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829847.937898:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829848.769824:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829848.961864:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829849.793833:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829849.985929:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829850.817819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829851.009928:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829851.841826:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829852.033936:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829852.353839:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829852.353850:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829852.353855:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829852.353866:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6489 00000100:00080000:1.0:1787829852.353878:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829852.353885:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829852.353930:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790800000/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829852.353957:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829852.353960:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829852.353961:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829852.353964:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6489 00000100:00080000:1.0:1787829852.353967:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829852.353971:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829852.353981:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829852.353993:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790800128/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829852.354006:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829852.354006:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790800000:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829852.354017:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829852.354024:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829852.354029:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6489 00000100:00080000:1.0:1787829852.354031:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829852.354036:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829852.354042:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829852.354051:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790800256/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829852.354058:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829852.354061:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829852.354062:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829852.354065:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6489 00000100:00080000:1.0:1787829852.354069:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829852.354074:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829852.354079:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829852.354085:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829852.354090:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790800384/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829852.354098:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829852.354102:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829852.354104:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829852.354109:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6489 00000100:00080000:1.0:1787829852.354112:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829852.354115:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829852.354119:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829852.354132:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829852/real 1787829852] req@ffff9e6831aa8f00 x1874668790800000/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829852.354137:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790800512/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829852.354184:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829852.354190:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790800000/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829852.354194:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829852.354201:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829852.354213:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6489 00000100:00080000:1.0:1787829852.354221:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829852.354223:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829852.354227:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787829852.354230:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000008:00100000:1.0:1787829852.354237:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829852.354243:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790800000:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787829852.354257:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790800640/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829852.354271:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829852.354272:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790800128:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829852.354281:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829852.354286:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829852.354291:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6489 00000100:00080000:1.0:1787829852.354301:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829852.354305:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829852.354309:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787829852.354314:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829852.354326:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790800768/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829852.354329:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829852/real 1787829852] req@ffff9e6831aa9e00 x1874668790800128/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829852.354393:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829852.354399:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790800128/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829852.354400:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829852.354405:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829852.354417:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6489 00000100:00080000:1.0:1787829852.354419:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829852.354422:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829852.354425:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829852.354444:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790800896/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829852.354456:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829852.354462:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829852.354466:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829852.354470:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829852.354473:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829852.354474:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790800128:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829852.354489:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6489 00000100:00080000:1.0:1787829852.354494:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829852.354495:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790800256:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829852.354501:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829852.354512:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829852.354529:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829852.354530:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790801024/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829852.354541:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829852/real 1787829852] req@ffff9e6831aa8b40 x1874668790800256/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829852.354568:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790800256/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829852.354575:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829852.354578:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829852.354580:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790800256:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829852.354585:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790800384:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829852.354603:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829852.354609:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829852/real 1787829852] req@ffff9e6831aab480 x1874668790800384/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829852.354617:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790800384/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829852.354622:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829852.354624:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829852.354627:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790800384:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829852.354631:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790800512:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829852.354641:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829852.354646:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829852/real 1787829852] req@ffff9e6831aaa580 x1874668790800512/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829852.354653:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790800512/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829852.354659:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829852.354661:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829852.354662:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790800512:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829852.354674:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790800640:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829852.354684:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829852.354690:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829852/real 1787829852] req@ffff9e6831aaad00 x1874668790800640/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829852.354721:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790800640/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829852.354747:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829852.354750:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829852.354752:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790800640:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829852.354757:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790800768:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829852.354773:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829852.354783:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829852/real 1787829852] req@ffff9e6831aaa1c0 x1874668790800768/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829852.354795:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790800768/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829852.354804:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829852.354807:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829852.354810:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790800768:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829852.354817:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790800896:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829852.354839:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829852.354849:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829852/real 1787829852] req@ffff9e6831aa8780 x1874668790800896/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829852.354857:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790800896/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829852.354862:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829852.354864:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829852.354866:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790800896:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829852.354870:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790801024:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829852.354882:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829852.354888:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829852/real 1787829852] req@ffff9e6831aaa940 x1874668790801024/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829852.354895:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790801024/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829852.354901:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829852.354903:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829852.354905:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790801024:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829852.865836:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829853.057908:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829853.889893:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829854.081899:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829854.913828:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829855.107233:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829855.937795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829856.129859:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829856.961817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829857.153950:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829857.473833:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829857.473841:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829857.473844:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829857.473852:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6494 00000100:00080000:2.0:1787829857.473861:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829857.473867:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829857.473900:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790801152/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829857.473924:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829857.473926:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829857.473928:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829857.473931:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6494 00000100:00080000:2.0:1787829857.473933:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829857.473936:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829857.473944:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829857.473958:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790801152:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829857.473965:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790801280/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829857.473975:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829857.473977:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829857.473980:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829857.473982:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6494 00000100:00080000:2.0:1787829857.473985:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829857.473988:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829857.473991:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829857.474000:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790801408/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829857.474006:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829857.474009:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829857.474010:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829857.474016:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829857.474016:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6494 00000100:00080000:2.0:1787829857.474023:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829857.474026:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829857.474030:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829857.474039:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790801536/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829857.474045:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829857.474048:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829857.474050:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829857.474053:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6494 00000100:00080000:2.0:1787829857.474056:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829857.474060:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829857/real 1787829857] req@ffff9e6804420b40 x1874668790801152/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829857.474062:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829857.474068:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829857.474083:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790801664/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829857.474086:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790801152/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829857.474103:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829857.474107:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829857.474108:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829857.474109:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829857.474118:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829857.474119:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6494 00000100:00080000:2.0:1787829857.474126:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829857.474131:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790801152:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829857.474132:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829857.474138:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829857.474147:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790801792/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829857.474153:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829857.474155:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829857.474156:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829857.474160:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6494 00000100:00080000:2.0:1787829857.474162:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829857.474167:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829857.474168:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790801280:192.168.203.109@tcp:38:kworker.0 00000008:00100000:2.0:1787829857.474171:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829857.474180:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790801920/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829857.474188:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829857.474190:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829857.474192:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829857.474196:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6494 00000100:00080000:2.0:1787829857.474200:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829857.474202:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829857.474207:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829857.474214:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829857.474218:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829857/real 1787829857] req@ffff9e68044230c0 x1874668790801280/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829857.474231:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790802048/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829857.474232:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790801280/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829857.474265:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829857.474267:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829857.474270:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829857.474276:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829857.474277:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790801280:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829857.474279:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829857.474286:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6494 00000100:00100000:1.0:1787829857.474289:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790801408:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829857.474291:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829857.474298:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829857.474303:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829857.474308:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829857.474317:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790802176/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829857.474320:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829857/real 1787829857] req@ffff9e68044212c0 x1874668790801408/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829857.474345:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790801408/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829857.474351:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829857.474354:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829857.474356:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790801408:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829857.474371:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790801536:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829857.474384:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829857.474391:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829857/real 1787829857] req@ffff9e6804421e00 x1874668790801536/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829857.474403:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790801536/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829857.474408:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829857.474410:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829857.474412:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790801536:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829857.474416:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790801664:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829857.474425:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829857.474431:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829857/real 1787829857] req@ffff9e6804421680 x1874668790801664/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829857.474438:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790801664/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829857.474443:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829857.474446:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829857.474447:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790801664:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829857.474451:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790801792:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829857.474460:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829857.474466:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829857/real 1787829857] req@ffff9e6804422580 x1874668790801792/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829857.474472:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790801792/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829857.474483:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829857.474486:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829857.474489:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790801792:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829857.474508:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790801920:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829857.474530:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829857.474542:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829857/real 1787829857] req@ffff9e6804420f00 x1874668790801920/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829857.474550:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790801920/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829857.474555:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829857.474558:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829857.474560:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790801920:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829857.474563:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790802048:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829857.474576:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829857.474582:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829857/real 1787829857] req@ffff9e6804423480 x1874668790802048/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829857.474590:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790802048/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829857.474595:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829857.474596:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829857.474599:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790802048:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829857.474602:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790802176:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829857.474613:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829857.474623:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829857/real 1787829857] req@ffff9e6804420780 x1874668790802176/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829857.474634:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790802176/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829857.474643:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829857.474646:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829857.474650:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790802176:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829857.985812:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829858.177903:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829859.009828:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829859.201869:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829860.033809:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829860.225970:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829861.057817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829861.249848:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829862.081774:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829862.273849:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829862.593800:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829862.593809:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829862.593813:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829862.593820:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6499 00000100:00080000:3.0:1787829862.593830:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829862.593835:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829862.593869:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790802304/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829862.593896:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829862.593899:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829862.593901:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829862.593906:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6499 00000100:00080000:3.0:1787829862.593909:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829862.593913:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829862.593930:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829862.593945:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790802432/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829862.593951:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790802304:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829862.593965:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829862.593968:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829862.593969:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829862.593973:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6499 00000100:00080000:3.0:1787829862.593976:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829862.593979:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829862.593983:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829862.593992:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790802560/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829862.593999:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829862.594002:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829862.594003:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829862.594005:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6499 00000100:00080000:3.0:1787829862.594009:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829862.594012:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829862.594016:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829862.594020:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829862.594029:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790802688/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829862.594036:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829862.594038:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829862.594039:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829862.594043:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6499 00000100:00080000:3.0:1787829862.594046:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829862.594048:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829862.594052:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829862.594053:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829862/real 1787829862] req@ffff9e680a2d4780 x1874668790802304/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829862.594061:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790802816/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829862.594069:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829862.594074:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790802304/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829862.594074:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829862.594078:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829862.594085:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6499 00000100:00080000:3.0:1787829862.594091:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829862.594096:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829862.594096:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829862.594103:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829862.594105:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829862.594110:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790802304:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829862.594114:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790802944/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829862.594127:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829862.594129:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829862.594131:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829862.594133:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6499 00000100:00100000:1.0:1787829862.594137:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790802432:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829862.594137:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829862.594143:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829862.594147:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829862.594155:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790803072/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829862.594162:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829862.594165:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829862.594169:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829862.594170:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829862.594174:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6499 00000100:00100000:1.0:1787829862.594177:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829862/real 1787829862] req@ffff9e680a2d70c0 x1874668790802432/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829862.594179:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829862.594186:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829862.594193:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829862.594197:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790802432/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829862.594204:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790803200/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829862.594207:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829862.594215:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829862.594217:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829862.594220:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790802432:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829862.594222:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829862.594226:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829862.594231:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790802560:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829862.594231:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6499 00000100:00080000:3.0:1787829862.594237:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829862.594241:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829862.594244:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829862.594253:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829862.594255:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790803328/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829862.594264:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829862/real 1787829862] req@ffff9e680a2d6580 x1874668790802560/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829862.594273:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790802560/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829862.594280:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829862.594282:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829862.594284:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790802560:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829862.594288:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790802688:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829862.594299:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829862.594305:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829862/real 1787829862] req@ffff9e680a2d4000 x1874668790802688/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829862.594313:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790802688/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829862.594318:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829862.594320:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829862.594322:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790802688:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829862.594325:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790802816:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829862.594334:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829862.594340:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829862/real 1787829862] req@ffff9e680a2d43c0 x1874668790802816/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829862.594347:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790802816/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829862.594352:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829862.594353:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829862.594355:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790802816:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829862.594358:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790802944:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829862.594367:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829862.594373:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829862/real 1787829862] req@ffff9e680a2d4f00 x1874668790802944/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829862.594379:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790802944/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829862.594390:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829862.594392:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829862.594394:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790802944:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829862.594407:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790803072:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829862.594417:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829862.594423:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829862/real 1787829862] req@ffff9e680a2d5a40 x1874668790803072/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829862.594430:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790803072/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829862.594435:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829862.594436:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829862.594438:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790803072:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829862.594441:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790803200:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829862.594451:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829862.594457:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829862/real 1787829862] req@ffff9e680a2d6d00 x1874668790803200/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829862.594464:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790803200/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829862.594469:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829862.594470:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829862.594473:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790803200:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829862.594476:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790803328:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829862.594486:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829862.594491:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829862/real 1787829862] req@ffff9e680a2d5e00 x1874668790803328/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829862.594498:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790803328/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829862.594512:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829862.594514:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829862.594516:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790803328:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829863.105814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829863.297836:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829864.129761:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829864.321841:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829865.153825:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829865.345875:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829866.177791:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829866.369856:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829867.201807:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829867.393874:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829867.713816:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829867.713824:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829867.713828:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829867.713837:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6504 00000100:00080000:3.0:1787829867.713848:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829867.713856:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829867.713894:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790803456/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829867.713918:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829867.713921:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829867.713923:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829867.713926:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6504 00000100:00080000:3.0:1787829867.713928:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829867.713940:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829867.713949:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829867.713959:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790803584/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829867.713966:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829867.713969:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829867.713970:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829867.713973:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6504 00000100:00080000:3.0:1787829867.713975:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829867.713980:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829867.713982:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790803456:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787829867.713984:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829867.713993:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790803712/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829867.714001:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829867.714003:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829867.714004:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829867.714006:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6504 00000100:00080000:3.0:1787829867.714008:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829867.714010:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829867.714013:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829867.714022:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790803840/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829867.714028:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829867.714031:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829867.714032:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829867.714035:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6504 00000100:00080000:3.0:1787829867.714037:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829867.714040:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829867.714043:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829867.714050:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790803968/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829867.714059:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829867.714060:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829867.714066:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829867.714067:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829867.714070:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6504 00000100:00080000:3.0:1787829867.714073:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829867.714076:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829867.714079:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829867.714087:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790804096/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829867.714092:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829867.714095:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829867.714096:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829867.714099:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6504 00000100:00080000:3.0:1787829867.714101:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829867.714104:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829867.714108:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829867.714114:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829867/real 1787829867] req@ffff9e680a2d7c00 x1874668790803456/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829867.714116:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790804224/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829867.714153:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829867.714156:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829867.714157:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829867.714160:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6504 00000100:00080000:3.0:1787829867.714162:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829867.714166:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790803456/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829867.714167:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829867.714172:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829867.714181:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790804352/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829867.714188:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829867.714189:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829867.714195:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829867.714196:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829867.714198:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829867.714203:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790803456:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829867.714203:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6504 00000100:00080000:3.0:1787829867.714209:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829867.714212:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829867.714216:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790803584:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787829867.714217:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829867.714231:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790804480/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829867.714249:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829867.714266:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829867/real 1787829867] req@ffff9e680a2d4b40 x1874668790803584/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829867.714280:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790803584/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829867.714290:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829867.714294:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829867.714311:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790803584:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829867.714339:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790803712:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829867.714364:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829867.714377:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829867/real 1787829867] req@ffff9e680a2d61c0 x1874668790803712/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829867.714390:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790803712/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829867.714398:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829867.714402:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829867.714406:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790803712:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829867.714414:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790803840:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829867.714437:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829867.714449:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829867/real 1787829867] req@ffff9e680a2d52c0 x1874668790803840/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829867.714470:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790803840/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829867.714478:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829867.714482:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829867.714485:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790803840:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829867.714491:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790803968:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829867.714512:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829867.714536:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829867/real 1787829867] req@ffff9e680a2d7480 x1874668790803968/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829867.714554:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790803968/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829867.714563:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829867.714567:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829867.714570:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790803968:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829867.714577:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790804096:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829867.714598:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829867.714611:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829867/real 1787829867] req@ffff9e680a2d5e00 x1874668790804096/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829867.714639:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790804096/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829867.714651:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829867.714654:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829867.714656:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790804096:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829867.714660:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790804224:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829867.714677:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829867.714685:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829867/real 1787829867] req@ffff9e680a2d6d00 x1874668790804224/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829867.714692:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790804224/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829867.714724:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829867.714727:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829867.714731:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790804224:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829867.714738:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790804352:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829867.714761:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829867.714776:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829867/real 1787829867] req@ffff9e680a2d5a40 x1874668790804352/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829867.714787:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790804352/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829867.714796:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829867.714799:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829867.714803:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790804352:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829867.714809:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790804480:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829867.714833:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829867.714857:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829867/real 1787829867] req@ffff9e680a2d4f00 x1874668790804480/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829867.714872:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790804480/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829867.714881:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829867.714884:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829867.714887:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790804480:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829868.225810:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829868.417852:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829869.249820:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829869.441917:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829870.273816:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829870.465856:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829871.297821:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829871.489879:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829872.321784:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829872.513908:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829872.833772:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829872.833783:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829872.833788:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829872.833798:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6509 00000100:00080000:3.0:1787829872.833808:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829872.833816:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829872.833853:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790804608/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829872.833884:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829872.833889:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829872.833891:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829872.833895:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6509 00000100:00080000:3.0:1787829872.833899:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829872.833904:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829872.833914:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829872.833928:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790804736/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829872.833940:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829872.833944:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787829872.833948:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790804608:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829872.833949:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829872.833958:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6509 00000100:00080000:3.0:1787829872.833962:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829872.833966:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829872.833971:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829872.833982:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790804864/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829872.833993:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829872.833997:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829872.833999:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829872.834003:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6509 00000100:00080000:3.0:1787829872.834007:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829872.834011:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1787829872.834017:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787829872.834018:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829872.834033:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790804992/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829872.834044:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829872.834048:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829872.834050:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829872.834054:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6509 00000100:00080000:3.0:1787829872.834058:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829872.834063:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829872.834068:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829872/real 1787829872] req@ffff9e680a2d43c0 x1874668790804608/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829872.834070:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829872.834094:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790805120/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829872.834095:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790804608/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829872.834123:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829872.834126:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829872.834128:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829872.834132:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829872.834135:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829872.834140:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6509 00000100:00100000:1.0:1787829872.834145:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790804608:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829872.834146:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829872.834160:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829872.834166:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829872.834180:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790804736:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829872.834182:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790805248/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829872.834197:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829872.834201:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829872.834204:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829872.834208:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6509 00000100:00080000:3.0:1787829872.834214:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829872.834216:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829872.834221:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829872.834228:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829872.834238:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829872/real 1787829872] req@ffff9e680a2d4000 x1874668790804736/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829872.834243:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790805376/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829872.834268:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829872.834273:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790804736/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829872.834274:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829872.834279:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829872.834288:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6509 00000100:00080000:1.0:1787829872.834295:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829872.834295:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829872.834302:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787829872.834305:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:3.0:1787829872.834309:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829872.834313:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790804736:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829872.834324:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790804864:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829872.834331:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790805504/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829872.834343:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829872.834347:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829872.834349:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829872.834355:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829872.834359:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6509 00000100:00080000:3.0:1787829872.834365:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829872.834370:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829872.834375:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829872.834376:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829872/real 1787829872] req@ffff9e680a2d6580 x1874668790804864/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829872.834389:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790805632/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829872.834397:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790804864/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829872.834411:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829872.834415:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829872.834421:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790804864:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829872.834430:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790804992:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829872.834453:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829872.834465:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829872/real 1787829872] req@ffff9e680a2d70c0 x1874668790804992/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829872.834479:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790804992/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829872.834489:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829872.834492:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829872.834496:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790804992:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829872.834502:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790805120:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829872.834524:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829872.834536:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829872/real 1787829872] req@ffff9e680a2d4780 x1874668790805120/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829872.834549:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790805120/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829872.834558:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829872.834561:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829872.834565:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790805120:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829872.834584:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790805248:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829872.834626:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829872.834638:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829872/real 1787829872] req@ffff9e680a2d43c0 x1874668790805248/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829872.834651:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790805248/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829872.834669:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829872.834672:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829872.834676:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790805248:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829872.834682:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790805376:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829872.834728:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829872.834742:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829872/real 1787829872] req@ffff9e680a2d4f00 x1874668790805376/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829872.834754:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790805376/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829872.834763:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829872.834766:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829872.834770:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790805376:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829872.834777:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790805504:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829872.834798:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829872.834810:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829872/real 1787829872] req@ffff9e680a2d5a40 x1874668790805504/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829872.834824:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790805504/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829872.834833:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829872.834836:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829872.834840:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790805504:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829872.834846:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790805632:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829872.834867:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829872.834879:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829872/real 1787829872] req@ffff9e680a2d6d00 x1874668790805632/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829872.834892:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790805632/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829872.834901:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829872.834904:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829872.834907:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790805632:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829873.345842:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829873.537845:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829874.369830:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829874.561895:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829875.393786:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829875.585853:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829876.417790:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829876.609914:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829877.441785:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829877.633895:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829877.953833:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829877.953841:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829877.953845:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829877.953853:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6514 00000100:00080000:1.0:1787829877.953863:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829877.953869:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829877.953903:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790805760/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829877.953928:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829877.953931:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829877.953933:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829877.953937:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6514 00000100:00080000:1.0:1787829877.953945:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829877.953947:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829877.953957:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829877.953970:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790805888/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829877.953978:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829877.953981:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829877.953982:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829877.953986:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6514 00000100:00080000:1.0:1787829877.953989:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829877.953992:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829877.953996:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829877.953998:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790805760:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787829877.954008:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790806016/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829877.954016:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829877.954019:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829877.954021:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829877.954024:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6514 00000100:00080000:1.0:1787829877.954026:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829877.954029:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829877.954032:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829877.954045:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790806144/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829877.954052:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829877.954055:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829877.954058:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829877.954064:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6514 00000100:00080000:1.0:1787829877.954067:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829877.954069:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829877.954072:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829877.954077:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829877.954084:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790806272/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829877.954091:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829877.954093:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829877.954094:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829877.954098:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6514 00000100:00080000:1.0:1787829877.954100:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829877.954102:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829877.954105:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829877.954112:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790806400/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829877.954119:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829877.954121:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829877.954122:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829877.954128:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6514 00000100:00100000:2.0:1787829877.954128:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829877/real 1787829877] req@ffff9e6831aabc00 x1874668790805760/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829877.954133:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829877.954138:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829877.954142:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829877.954152:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790806528/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829877.954159:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829877.954162:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787829877.954163:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790805760/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829877.954164:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829877.954171:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6514 00000100:00080000:1.0:1787829877.954174:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829877.954180:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829877.954185:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829877.954192:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787829877.954196:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790806656/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829877.954202:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829877.954207:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829877.954211:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787829877.954212:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790805760:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829877.954213:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829877.954220:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6514 00000100:00080000:1.0:1787829877.954223:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829877.954225:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829877.954229:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790805888:192.168.203.109@tcp:38:kworker.0 00000008:00100000:1.0:1787829877.954237:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829877.954247:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790806784/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829877.954263:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829877.954283:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829877/real 1787829877] req@ffff9e6831aa9a40 x1874668790805888/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829877.954300:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790805888/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829877.954312:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829877.954316:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829877.954320:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790805888:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829877.954366:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790806016:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829877.954395:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829877.954415:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829877/real 1787829877] req@ffff9e6831aa8000 x1874668790806016/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829877.954433:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790806016/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829877.954444:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829877.954449:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829877.954454:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790806016:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829877.954464:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790806144:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829877.954494:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829877.954515:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829877/real 1787829877] req@ffff9e6831aa9680 x1874668790806144/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829877.954534:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790806144/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829877.954545:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829877.954566:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829877.954573:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790806144:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829877.954582:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790806272:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829877.954611:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829877.954627:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829877/real 1787829877] req@ffff9e6831aaa940 x1874668790806272/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829877.954647:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790806272/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829877.954656:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829877.954660:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829877.954664:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790806272:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829877.954671:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790806400:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829877.954729:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829877.954746:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829877/real 1787829877] req@ffff9e6831aa8780 x1874668790806400/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829877.954763:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790806400/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829877.954775:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829877.954780:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829877.954784:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790806400:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829877.954791:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790806528:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829877.954824:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829877.954838:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829877/real 1787829877] req@ffff9e6831aaa1c0 x1874668790806528/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829877.954852:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790806528/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829877.954861:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829877.954864:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829877.954868:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790806528:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829877.954875:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790806656:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829877.954899:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829877.954912:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829877/real 1787829877] req@ffff9e6831aaad00 x1874668790806656/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829877.954925:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790806656/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829877.954933:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829877.954937:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829877.954941:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790806656:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829877.954947:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790806784:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829877.954971:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829877.954983:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829877/real 1787829877] req@ffff9e6831aaa580 x1874668790806784/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829877.954996:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790806784/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829877.955005:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829877.955009:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829877.955013:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790806784:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829878.465831:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829878.657953:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829879.489783:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829879.681877:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829880.513822:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829880.705870:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829881.537773:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829881.729868:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829882.561819:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829882.753908:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829883.073786:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829883.073797:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829883.073802:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829883.073813:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6519 00000100:00080000:1.0:1787829883.073822:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829883.073828:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829883.073860:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790806912/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829883.073883:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829883.073886:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829883.073887:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829883.073890:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6519 00000100:00080000:1.0:1787829883.073892:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829883.073895:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829883.073904:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829883.073914:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790807040/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829883.073923:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829883.073925:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790806912:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829883.073926:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829883.073928:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829883.073936:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6519 00000100:00080000:1.0:1787829883.073938:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829883.073941:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829883.073944:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829883.073951:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790807168/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829883.073965:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829883.073968:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829883.073969:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829883.073972:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6519 00000100:00080000:1.0:1787829883.073975:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829883.073977:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829883.073981:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829883.073990:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790807296/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829883.073998:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829883.074001:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829883.074004:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829883.074006:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829883.074009:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6519 00000100:00080000:1.0:1787829883.074012:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829883.074015:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829883.074018:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829883.074025:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790807424/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829883.074032:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829883.074035:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829883.074036:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829883.074039:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6519 00000100:00080000:1.0:1787829883.074042:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829883.074048:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829883.074049:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829883/real 1787829883] req@ffff9e6831aab480 x1874668790806912/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829883.074053:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829883.074070:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790807552/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829883.074077:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790806912/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829883.074085:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829883.074094:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829883.074098:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829883.074102:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829883.074103:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6519 00000100:00080000:1.0:1787829883.074111:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829883.074114:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829883.074117:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829883.074123:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829883.074124:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790806912:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787829883.074134:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790807680/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829883.074145:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829883.074149:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829883.074151:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829883.074156:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790807040:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829883.074158:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6519 00000100:00080000:1.0:1787829883.074166:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829883.074171:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829883.074177:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829883.074188:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829883.074193:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790807808/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829883.074203:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829883.074206:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829883/real 1787829883] req@ffff9e6831aa8b40 x1874668790807040/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829883.074207:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829883.074211:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829883.074218:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6519 00000100:00080000:1.0:1787829883.074224:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829883.074228:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790807040/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829883.074229:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829883.074236:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829883.074241:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829883.074246:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829883.074249:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790807936/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829883.074253:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790807040:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829883.074266:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790807168:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829883.074283:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829883.074290:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829883/real 1787829883] req@ffff9e6831aa9e00 x1874668790807168/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829883.074298:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790807168/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829883.074303:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829883.074305:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829883.074307:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790807168:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829883.074310:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790807296:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829883.074320:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829883.074325:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829883/real 1787829883] req@ffff9e6831aa8f00 x1874668790807296/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829883.074332:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790807296/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829883.074338:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829883.074340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829883.074342:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790807296:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829883.074345:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790807424:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829883.074354:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829883.074360:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829883/real 1787829883] req@ffff9e6831aaa580 x1874668790807424/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829883.074366:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790807424/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829883.074371:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829883.074372:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829883.074374:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790807424:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829883.074377:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790807552:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829883.074386:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829883.074392:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829883/real 1787829883] req@ffff9e6831aaad00 x1874668790807552/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829883.074399:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790807552/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829883.074405:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829883.074406:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829883.074409:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790807552:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829883.074412:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790807680:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829883.074422:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829883.074428:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829883/real 1787829883] req@ffff9e6831aaa1c0 x1874668790807680/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829883.074434:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790807680/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829883.074439:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829883.074441:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829883.074443:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790807680:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829883.074455:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790807808:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829883.074465:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829883.074471:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829883/real 1787829883] req@ffff9e6831aa8780 x1874668790807808/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829883.074477:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790807808/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829883.074482:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829883.074484:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829883.074485:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790807808:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829883.074489:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790807936:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829883.074498:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829883.074504:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829883/real 1787829883] req@ffff9e6831aaa940 x1874668790807936/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829883.074511:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790807936/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829883.074530:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829883.074532:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829883.074535:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790807936:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829883.585769:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829883.777942:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829884.609796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829884.801923:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829885.633818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829885.825861:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829886.657808:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829886.849930:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829887.681844:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829887.873854:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829888.193848:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829888.193860:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829888.193864:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829888.193874:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6524 00000100:00080000:2.0:1787829888.193887:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829888.193895:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829888.193935:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790808064/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829888.193972:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829888.193977:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829888.193979:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829888.193983:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6524 00000100:00080000:2.0:1787829888.193989:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829888.193994:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829888.194004:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829888.194023:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790808192/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829888.194028:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790808064:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829888.194053:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829888.194057:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829888.194059:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829888.194067:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6524 00000100:00080000:2.0:1787829888.194072:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829888.194077:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829888.194083:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829888.194098:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790808320/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829888.194110:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829888.194114:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829888.194116:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829888.194121:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6524 00000800:00000100:1.0:1787829888.194127:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829888.194130:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829888.194139:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829888.194147:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829888.194171:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829888/real 1787829888] req@ffff9e6804422d00 x1874668790808064/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829888.194176:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790808448/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829888.194198:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790808064/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829888.194205:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829888.194216:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829888.194216:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829888.194223:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829888.194227:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790808064:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829888.194231:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829888.194239:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6524 00000100:00080000:2.0:1787829888.194246:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829888.194251:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829888.194255:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790808192:192.168.203.109@tcp:38:kworker.0 00000008:00100000:2.0:1787829888.194260:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829888.194280:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829888.194287:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790808576/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829888.194297:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829888/real 1787829888] req@ffff9e6804423840 x1874668790808192/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829888.194313:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829888.194316:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790808192/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829888.194328:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829888.194336:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829888.194336:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829888.194345:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829888.194350:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790808192:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829888.194350:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6524 00000100:00100000:1.0:1787829888.194364:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790808320:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829888.194366:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829888.194377:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1787829888.194383:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:2.0:1787829888.194384:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829888.194393:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829888/real 1787829888] req@ffff9e68044221c0 x1874668790808320/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829888.194405:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790808704/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829888.194406:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790808320/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829888.194490:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829888.194495:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829888.194498:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829888.194501:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790808320:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829888.194511:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787829888.194513:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790808448:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787829888.194516:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829888.194529:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6524 00000100:00080000:2.0:1787829888.194534:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829888.194541:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1787829888.194548:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:2.0:1787829888.194549:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829888.194559:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829888/real 1787829888] req@ffff9e6804422940 x1874668790808448/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829888.194571:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790808832/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829888.194574:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790808448/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829888.194602:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829888.194607:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829888.194615:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790808448:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787829888.194640:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829888.194642:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790808576:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787829888.194653:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829888.194657:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829888.194665:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829888.194666:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6524 00000100:00080000:2.0:1787829888.194675:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829888.194676:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829888/real 1787829888] req@ffff9e6804422d00 x1874668790808576/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829888.194691:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829888.194713:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790808576/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829888.194721:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829888.194729:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829888.194733:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790808576:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829888.194738:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790808704:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829888.194754:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829888.194763:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829888/real 1787829888] req@ffff9e6804420780 x1874668790808704/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829888.194764:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829888.194791:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790808704/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829888.194797:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790808960/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829888.194807:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829888.194821:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829888.194833:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790808704:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829888.194845:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829888.194846:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790808832:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829888.194859:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829888.194862:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829888.194870:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829888.194872:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6524 00000100:00080000:2.0:1787829888.194880:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829888.194881:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829888/real 1787829888] req@ffff9e6804423480 x1874668790808832/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829888.194896:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829888.194898:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790808832/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829888.194911:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787829888.194914:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829888.194916:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829888.194923:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790808832:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829888.194941:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790809088/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829888.194943:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790808960:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829888.194979:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829888.194994:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829888/real 1787829888] req@ffff9e6804420f00 x1874668790808960/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829888.195005:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790808960/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829888.195019:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829888.195022:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829888.195025:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790808960:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829888.195038:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790809088:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829888.195057:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829888.195068:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829888/real 1787829888] req@ffff9e6804422580 x1874668790809088/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829888.195079:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790809088/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829888.195086:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829888.195089:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829888.195092:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790809088:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829888.705826:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829888.897896:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829889.729801:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829889.921899:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829890.753808:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829890.945883:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829891.777803:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829891.969938:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829892.801821:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829892.993865:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829893.314000:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829893.314010:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829893.314015:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829893.314033:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6530 00000100:00080000:1.0:1787829893.314044:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829893.314052:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829893.314100:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9680 x1874668790809216/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829893.314125:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829893.314131:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829893.314132:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829893.314136:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6530 00000100:00080000:1.0:1787829893.314139:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829893.314143:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829893.314152:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829893.314163:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8000 x1874668790809344/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829893.314167:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790809216:192.168.203.109@tcp:250:kworker.0 00000100:00080000:1.0:1787829893.314179:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829893.314182:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829893.314183:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829893.314186:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6530 00000100:00080000:1.0:1787829893.314188:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829893.314191:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829893.314194:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829893.314203:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9a40 x1874668790809472/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829893.314210:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829893.314212:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829893.314213:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829893.314216:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6530 00000100:00080000:1.0:1787829893.314221:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829893.314224:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829893.314227:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829893.314230:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829893.314239:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aabc00 x1874668790809600/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829893.314247:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829893.314250:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829893.314251:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829893.314254:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6530 00000100:00080000:1.0:1787829893.314256:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829893.314259:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829893.314263:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829893.314269:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829893/real 1787829893] req@ffff9e6831aa9680 x1874668790809216/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829893.314274:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa940 x1874668790809728/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829893.314305:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829893.314310:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829893.314313:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829893.314318:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9680 x1874668790809216/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829893.314319:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6530 00000100:00080000:1.0:1787829893.314327:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829893.314334:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787829893.314335:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787829893.314342:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829893.314342:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829893.314347:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790809216:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787829893.314361:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8780 x1874668790809856/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829893.314370:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790809344:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829893.314386:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829893.314389:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829893.314391:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829893.314395:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6530 00000800:00000100:2.0:1787829893.314399:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829893.314402:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829893.314408:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829893.314413:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829893.314414:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829893/real 1787829893] req@ffff9e6831aa8000 x1874668790809344/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829893.314425:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790809984/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829893.314428:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8000 x1874668790809344/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829893.314463:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829893.314467:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829893.314468:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829893.314472:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829893.314473:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829893.314479:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790809344:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829893.314484:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6530 00000100:00100000:2.0:1787829893.314488:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790809472:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829893.314500:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829893.314506:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787829893.314511:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1787829893.314514:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829893.314522:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829893/real 1787829893] req@ffff9e6831aa9a40 x1874668790809472/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829893.314539:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790810112/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829893.314539:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9a40 x1874668790809472/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829893.314576:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829893.314579:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829893.314583:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829893.314584:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:1.0:1787829893.314589:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829893.314589:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790809472:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829893.314601:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6530 00000100:00100000:2.0:1787829893.314602:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790809600:192.168.203.109@tcp:8:kworker.0 00000100:00080000:1.0:1787829893.314611:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829893.314619:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787829893.314624:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:1.0:1787829893.314628:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829893.314633:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829893/real 1787829893] req@ffff9e6831aabc00 x1874668790809600/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829893.314643:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790810240/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829893.314644:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aabc00 x1874668790809600/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829893.314673:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829893.314679:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829893.314682:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aabc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790809600:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829893.314687:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790809728:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829893.314720:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829893.314728:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829893/real 1787829893] req@ffff9e6831aaa940 x1874668790809728/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829893.314735:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa940 x1874668790809728/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829893.314741:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829893.314743:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829893.314745:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790809728:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829893.314757:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790809856:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829893.314769:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829893.314775:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829893/real 1787829893] req@ffff9e6831aa8780 x1874668790809856/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829893.314782:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8780 x1874668790809856/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829893.314788:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829893.314790:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829893.314792:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790809856:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829893.314795:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790809984:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829893.314805:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829893.314811:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829893/real 1787829893] req@ffff9e6831aaa1c0 x1874668790809984/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829893.314818:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790809984/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829893.314823:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829893.314825:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829893.314826:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790809984:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829893.314830:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790810112:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829893.314839:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829893.314845:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829893/real 1787829893] req@ffff9e6831aaad00 x1874668790810112/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829893.314852:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790810112/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829893.314857:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829893.314859:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829893.314861:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790810112:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829893.314864:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790810240:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829893.314873:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829893.314878:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829893/real 1787829893] req@ffff9e6831aaa580 x1874668790810240/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829893.314885:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790810240/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829893.314889:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829893.314891:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829893.314893:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790810240:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829893.825816:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829894.017915:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829894.849805:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829895.041884:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829895.873826:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829896.065906:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829896.897827:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829897.089850:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829897.921836:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829898.113893:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829898.433841:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829898.433852:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829898.433856:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829898.433866:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6535 00000100:00080000:3.0:1787829898.433877:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829898.433886:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829898.433934:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790810368/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829898.433966:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829898.433971:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829898.433973:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829898.433977:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6535 00000100:00080000:3.0:1787829898.433981:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829898.433985:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829898.433995:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829898.434009:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790810496/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829898.434020:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829898.434024:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829898.434025:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829898.434036:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6535 00000100:00080000:3.0:1787829898.434040:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829898.434044:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829898.434048:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829898.434060:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790810624/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829898.434069:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829898.434072:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829898.434074:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829898.434078:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6535 00000100:00080000:3.0:1787829898.434082:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829898.434093:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829898.434099:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829898.434111:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790810752/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829898.434120:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829898.434123:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829898.434125:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829898.434129:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6535 00000100:00080000:3.0:1787829898.434132:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829898.434136:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829898.434142:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829898.434160:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790810880/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829898.434171:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829898.434174:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829898.434176:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829898.434179:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6535 00000100:00080000:3.0:1787829898.434183:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829898.434187:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829898.434191:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829898.434212:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790811008/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829898.434221:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829898.434225:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829898.434227:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829898.434231:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6535 00000100:00080000:3.0:1787829898.434234:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829898.434238:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829898.434242:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829898.434254:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790811136/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829898.434263:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829898.434266:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829898.434268:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829898.434272:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6535 00000100:00080000:3.0:1787829898.434276:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829898.434280:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829898.434285:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829898.434296:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790811264/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829898.434306:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829898.434309:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829898.434311:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829898.434315:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6535 00000100:00080000:3.0:1787829898.434318:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829898.434322:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829898.434326:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829898.434336:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790811392/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829898.434481:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790810368:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829898.434562:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790810496:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829898.434582:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790810624:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829898.434596:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790810752:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829898.434613:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790810880:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829898.434625:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790811008:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829898.434640:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790811136:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829898.434652:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790811264:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829898.434664:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790811392:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829898.945765:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829899.137885:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829899.969837:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829900.161856:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829900.993760:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829901.185898:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787829901.506083:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:1.0:1787829901.506121:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787829901.506144:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829901.506188:0:2387:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:1.0:1787829901.506199:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829901.506207:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829901.506214:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829901.506221:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829901.506228:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829901.506234:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787829901.506240:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:2.0:1787829901.506545:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829898/real 1787829901] req@ffff9e680a2d5e00 x1874668790810368/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829901.506568:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790810368/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829901.506579:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829901.506583:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829901.506587:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790810368:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829901.506594:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829898/real 1787829901] req@ffff9e680a2d7480 x1874668790810496/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829901.506606:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790810496/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829901.506614:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829901.506617:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829901.506620:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790810496:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829901.506628:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829898/real 1787829901] req@ffff9e680a2d52c0 x1874668790810624/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829901.506639:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790810624/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829901.506646:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829901.506649:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829901.506652:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790810624:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829901.506658:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829898/real 1787829901] req@ffff9e680a2d61c0 x1874668790810752/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829901.506668:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790810752/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829901.506675:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829901.506678:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829901.506681:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790810752:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829901.506687:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829898/real 1787829901] req@ffff9e680a2d4b40 x1874668790810880/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829901.506727:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790810880/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829901.506735:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829901.506738:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829901.506741:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790810880:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829901.506748:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829898/real 1787829901] req@ffff9e680a2d7c00 x1874668790811008/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829901.506758:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790811008/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829901.506767:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829901.506770:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829901.506773:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790811008:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829901.506778:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829898/real 1787829901] req@ffff9e680a2d6d00 x1874668790811136/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829901.507349:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790811136/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829901.507359:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829901.507362:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829901.507366:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790811136:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829901.507380:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829898/real 1787829901] req@ffff9e680a2d5a40 x1874668790811264/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829901.507402:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790811264/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829901.507410:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829901.507414:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829901.507417:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790811264:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829901.507423:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829898/real 1787829901] req@ffff9e680a2d4f00 x1874668790811392/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829901.507444:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790811392/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829901.507452:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829901.507454:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829901.507457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790811392:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829902.017795:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829902.209852:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829903.041820:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829903.233880:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829903.553792:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829903.553805:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829903.553811:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829903.553823:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6540 00000100:00080000:3.0:1787829903.553837:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829903.553846:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829903.553889:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790811520/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829903.553917:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829903.553921:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829903.553923:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829903.553936:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6540 00000100:00080000:3.0:1787829903.553941:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829903.553947:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829903.553959:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829903.553973:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790811520:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829903.553979:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790811648/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829903.553995:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829903.554000:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829903.554003:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829903.554009:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6540 00000100:00080000:3.0:1787829903.554014:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829903.554021:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829903.554027:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829903.554041:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790811776/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829903.554052:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829903.554056:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829903.554060:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829903.554062:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829903.554074:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6540 00000100:00080000:3.0:1787829903.554083:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829903.554089:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829903.554096:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829903.554113:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790811904/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829903.554124:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829903.554126:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829903/real 1787829903] req@ffff9e680a2d43c0 x1874668790811520/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829903.554130:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829903.554134:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829903.554158:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6540 00000100:00100000:2.0:1787829903.554160:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790811520/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829903.554166:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829903.554177:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:2.0:1787829903.554184:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787829903.554185:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829903.554196:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829903.554204:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790811520:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829903.554207:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790812032/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829903.554228:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829903.554232:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829903.554234:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829903.554239:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6540 00000100:00080000:3.0:1787829903.554243:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829903.554249:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829903.554254:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790811648:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787829903.554257:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829903.554285:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790812160/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829903.554299:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829903.554303:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829903.554305:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829903.554311:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6540 00000100:00080000:3.0:1787829903.554317:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829903.554323:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829903.554325:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829903.554335:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829903.554345:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829903/real 1787829903] req@ffff9e680a2d4780 x1874668790811648/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829903.554354:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790812288/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829903.554366:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790811648/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829903.554382:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829903.554387:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829903.554393:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829903.554403:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6540 00000100:00080000:2.0:1787829903.554404:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829903.554409:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829903.554416:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829903.554423:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829903.554432:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790811648:192.168.203.109@tcp:38:kworker.0 00000008:00100000:3.0:1787829903.554436:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829903.554447:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790811776:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829903.554458:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790812416/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829903.554470:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829903.554474:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829903.554476:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829903.554483:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6540 00000800:00000100:2.0:1787829903.554486:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829903.554492:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829903.554499:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829903.554506:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829903.554513:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829903/real 1787829903] req@ffff9e680a2d70c0 x1874668790811776/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829903.554525:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790812544/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829903.554537:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790811776/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829903.554560:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829903.554565:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829903.554568:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790811776:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829903.554579:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790811904:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829903.554602:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829903.554613:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829903/real 1787829903] req@ffff9e680a2d6580 x1874668790811904/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829903.554623:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790811904/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829903.554630:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829903.554632:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829903.554636:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790811904:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829903.554642:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790812032:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829903.554660:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829903.554671:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829903/real 1787829903] req@ffff9e680a2d4000 x1874668790812032/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829903.554681:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790812032/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829903.554689:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829903.554692:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829903.554733:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790812032:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829903.554755:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790812160:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829903.554776:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829903.554786:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829903/real 1787829903] req@ffff9e680a2d43c0 x1874668790812160/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829903.554797:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790812160/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829903.554806:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829903.554809:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829903.554813:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790812160:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829903.554821:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790812288:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829903.554837:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829903.554846:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829903/real 1787829903] req@ffff9e680a2d4f00 x1874668790812288/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829903.554855:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790812288/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829903.554861:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829903.554864:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829903.554867:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790812288:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829903.554873:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790812416:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829903.554891:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829903.554903:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829903/real 1787829903] req@ffff9e680a2d5a40 x1874668790812416/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829903.554911:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790812416/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829903.554917:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829903.554920:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829903.554923:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790812416:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829903.554929:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790812544:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829903.554945:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829903.554953:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829903/real 1787829903] req@ffff9e680a2d6d00 x1874668790812544/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829903.554961:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790812544/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829903.554968:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829903.554970:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829903.554973:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790812544:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829904.065855:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829904.257925:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829905.089800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829905.281876:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829906.113827:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829906.305881:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829907.137811:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829907.329852:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829908.161800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829908.353929:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829908.673825:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829908.673834:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829908.673839:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829908.673849:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6545 00000100:00080000:3.0:1787829908.673864:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829908.673871:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829908.673905:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790812672/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829908.673930:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829908.673933:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829908.673935:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829908.673939:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6545 00000100:00080000:3.0:1787829908.673944:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829908.673949:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829908.673958:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829908.673962:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790812672:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829908.673972:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790812800/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829908.673981:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829908.673983:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829908.673984:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829908.673987:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6545 00000100:00080000:3.0:1787829908.673990:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829908.673992:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829908.673996:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829908.674003:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790812928/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829908.674009:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829908.674011:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829908.674013:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829908.674016:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829908.674019:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6545 00000100:00080000:3.0:1787829908.674023:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829908.674026:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829908.674030:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829908.674037:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790813056/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829908.674042:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829908.674045:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829908.674046:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829908.674048:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6545 00000100:00080000:3.0:1787829908.674051:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829908.674053:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829908/real 1787829908] req@ffff9e680a2d7c00 x1874668790812672/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829908.674056:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829908.674061:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829908.674070:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790813184/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829908.674072:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790812672/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829908.674080:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829908.674087:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829908.674091:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829908.674093:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790812672:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829908.674095:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829908.674098:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829908.674102:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6545 00000100:00080000:3.0:1787829908.674104:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829908.674107:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829908.674109:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829908.674115:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790812800:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829908.674117:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790813312/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829908.674126:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829908.674128:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829908.674130:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829908.674136:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829908.674137:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6545 00000100:00080000:3.0:1787829908.674144:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829908.674149:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829908/real 1787829908] req@ffff9e680a2d4b40 x1874668790812800/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829908.674150:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829908.674160:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829908.674164:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790812800/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829908.674171:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829908.674175:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829908.674176:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790813440/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829908.674178:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790812800:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829908.674187:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790812928:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829908.674196:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829908.674200:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1787829908.674203:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829908.674204:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829908.674212:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829908/real 1787829908] req@ffff9e680a2d61c0 x1874668790812928/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829908.674212:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6545 00000100:00080000:3.0:1787829908.674220:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829908.674224:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790812928/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829908.674226:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829908.674234:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829908.674235:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829908.674240:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829908.674242:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790812928:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829908.674246:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790813568/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829908.674248:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790813056:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829908.674260:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829908.674263:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829908.674264:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787829908.674267:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829908.674269:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6545 00000100:00080000:3.0:1787829908.674274:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829908.674276:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829908.674279:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829908/real 1787829908] req@ffff9e680a2d52c0 x1874668790813056/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829908.674282:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829908.674290:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790813056/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829908.674296:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829908.674300:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829908.674300:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790813696/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829908.674305:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790813056:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829908.674312:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790813184:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829908.674329:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829908.674349:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829908/real 1787829908] req@ffff9e680a2d7480 x1874668790813184/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829908.674358:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790813184/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829908.674363:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829908.674365:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829908.674367:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790813184:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829908.674377:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790813312:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829908.674390:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829908.674396:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829908/real 1787829908] req@ffff9e680a2d5e00 x1874668790813312/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829908.674403:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790813312/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829908.674408:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829908.674410:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829908.674412:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790813312:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829908.674415:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790813440:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829908.674423:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829908.674428:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829908/real 1787829908] req@ffff9e680a2d7c00 x1874668790813440/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829908.674434:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790813440/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829908.674442:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829908.674444:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829908.674446:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790813440:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829908.674449:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790813568:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829908.674457:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829908.674462:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829908/real 1787829908] req@ffff9e680a2d6d00 x1874668790813568/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829908.674467:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790813568/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829908.674472:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829908.674474:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829908.674475:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790813568:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829908.674478:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790813696:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829908.674485:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829908.674490:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829908/real 1787829908] req@ffff9e680a2d5a40 x1874668790813696/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829908.674496:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790813696/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829908.674500:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829908.674502:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829908.674504:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790813696:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829909.185848:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829909.377880:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829910.209834:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829910.401911:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829911.233796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829911.425952:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829912.257862:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829912.449855:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829913.281851:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829913.473869:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829913.793800:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829913.793810:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829913.793815:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829913.793826:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6550 00000100:00080000:3.0:1787829913.793835:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829913.793842:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829913.793876:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790813824/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829913.793905:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829913.793909:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829913.793911:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829913.793916:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6550 00000100:00080000:3.0:1787829913.793919:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829913.793924:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829913.793933:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829913.793937:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790813824:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829913.793949:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790813952/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829913.793960:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829913.793963:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829913.793965:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829913.793968:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6550 00000100:00080000:3.0:1787829913.793972:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829913.793976:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829913.793980:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829913.793992:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790814080/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829913.793996:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829913.794011:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829913.794014:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829913.794016:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829913.794020:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6550 00000100:00080000:3.0:1787829913.794024:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829913.794027:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829913.794032:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829913.794042:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790814208/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829913.794049:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829913/real 1787829913] req@ffff9e680a2d4f00 x1874668790813824/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829913.794064:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790813824/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829913.794075:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829913.794080:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829913.794083:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790813824:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829913.794116:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829913.794117:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790813952:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829913.794123:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829913.794126:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829913.794131:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6550 00000100:00080000:3.0:1787829913.794136:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829913.794140:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829913.794146:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829913.794147:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829913.794164:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829913/real 1787829913] req@ffff9e680a2d43c0 x1874668790813952/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829913.794164:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790814336/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829913.794193:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790813952/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829913.794201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829913.794204:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829913.794207:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790813952:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829913.794213:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790814080:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829913.794233:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829913.794244:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829913/real 1787829913] req@ffff9e680a2d4000 x1874668790814080/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829913.794262:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790814080/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829913.794270:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829913.794272:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829913.794274:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790814080:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829913.794286:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790814208:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829913.794317:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829913.794322:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829913.794324:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829913.794328:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829913.794334:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6550 00000100:00100000:1.0:1787829913.794339:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829913/real 1787829913] req@ffff9e680a2d6580 x1874668790814208/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829913.794341:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829913.794357:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790814208/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829913.794364:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829913.794367:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829913.794371:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790814208:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829913.794384:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790814336:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829913.794403:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829913.794423:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829913/real 1787829913] req@ffff9e680a2d70c0 x1874668790814336/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829913.794431:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790814336/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829913.794441:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829913.794443:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829913.794445:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790814336:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829913.794451:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829913.794459:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829913.794474:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790814464/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829913.794490:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829913.794493:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829913.794494:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829913.794498:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6550 00000100:00080000:3.0:1787829913.794503:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829913.794507:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829913.794512:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829913.794517:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790814464:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829913.794524:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790814592/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829913.794535:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829913.794538:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829913.794540:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829913.794546:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829913.794549:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6550 00000100:00080000:3.0:1787829913.794555:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829913.794559:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829913.794564:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829913/real 1787829913] req@ffff9e680a2d4780 x1874668790814464/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829913.794566:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829913.794581:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790814720/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829913.794585:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790814464/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829913.794611:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829913.794615:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829913.794616:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829913.794621:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790814464:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829913.794625:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829913.794628:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829913.794634:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6550 00000100:00100000:1.0:1787829913.794635:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790814592:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829913.794642:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829913.794648:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829913.794653:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829913.794657:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829913.794664:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790814848/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829913.794669:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829913/real 1787829913] req@ffff9e680a2d70c0 x1874668790814592/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829913.794685:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790814592/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829913.794692:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829913.794714:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829913.794718:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790814592:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829913.794724:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790814720:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829913.794744:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829913.794754:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829913/real 1787829913] req@ffff9e680a2d6580 x1874668790814720/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829913.794764:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790814720/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829913.794771:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829913.794773:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829913.794775:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790814720:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829913.794784:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790814848:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829913.794797:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829913.794803:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829913/real 1787829913] req@ffff9e680a2d4000 x1874668790814848/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829913.794809:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790814848/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829913.794814:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829913.794816:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829913.794818:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790814848:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829914.305813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829914.497839:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829915.329808:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829915.521881:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829916.353800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829916.545884:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829917.377833:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829917.569841:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829918.401812:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829918.593834:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829918.913826:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829918.913837:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829918.913841:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829918.913850:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6555 00000100:00080000:2.0:1787829918.913860:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829918.913868:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829918.913905:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790814976/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829918.913930:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829918.913934:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829918.913936:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829918.913939:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6555 00000100:00080000:2.0:1787829918.913943:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829918.913947:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829918.913956:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829918.913968:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790814976:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829918.913971:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790815104/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829918.913983:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829918.913986:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829918.913997:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829918.914002:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6555 00000100:00080000:2.0:1787829918.914006:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829918.914011:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829918.914016:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829918.914027:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829918.914029:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790815232/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829918.914041:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829918.914045:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829918.914047:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829918.914051:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6555 00000100:00080000:2.0:1787829918.914056:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829918.914060:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829918.914064:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829918/real 1787829918] req@ffff9e6804421680 x1874668790814976/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829918.914067:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829918.914085:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790815360/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829918.914094:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790814976/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829918.914097:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829918.914106:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829918.914109:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829918.914116:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829918.914116:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6555 00000100:00080000:2.0:1787829918.914120:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829918.914124:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829918.914128:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829918.914134:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790814976:192.168.203.109@tcp:250:kworker.0 00000008:00100000:2.0:1787829918.914135:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829918.914150:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790815488/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829918.914156:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790815104:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829918.914165:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829918.914169:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829918.914171:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829918.914176:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6555 00000800:00000100:1.0:1787829918.914181:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829918.914181:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829918.914188:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829918.914194:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829918.914195:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829918/real 1787829918] req@ffff9e6804421e00 x1874668790815104/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829918.914206:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790815104/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829918.914211:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790815616/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829918.914214:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829918.914220:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829918.914226:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790815104:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829918.914233:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829918.914235:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790815232:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829918.914242:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829918.914245:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829918.914249:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6555 00000800:00000100:1.0:1787829918.914251:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829918.914257:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829918.914262:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829918/real 1787829918] req@ffff9e68044212c0 x1874668790815232/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829918.914280:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829918.914286:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829918.914304:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790815744/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829918.914305:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790815232/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829918.914330:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829918.914331:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829918.914338:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829918.914338:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829918.914344:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829918.914345:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790815232:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829918.914352:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6555 00000100:00100000:1.0:1787829918.914355:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790815360:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787829918.914360:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829918.914367:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829918.914373:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829918.914376:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829918.914386:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829918/real 1787829918] req@ffff9e68044230c0 x1874668790815360/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829918.914390:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790815872/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829918.914400:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790815360/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829918.914408:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829918.914410:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829918.914412:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790815360:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787829918.914413:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829918.914418:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829918.914420:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829918.914425:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6555 00000100:00100000:1.0:1787829918.914426:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790815488:192.168.203.109@tcp:8:kworker.0 00000100:00080000:2.0:1787829918.914432:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829918.914438:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1787829918.914442:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:2.0:1787829918.914444:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829918.914450:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829918/real 1787829918] req@ffff9e6804420b40 x1874668790815488/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829918.914458:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790816000/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829918.914459:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790815488/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829918.914490:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829918.914493:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829918.914496:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790815488:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829918.914499:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790815616:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829918.914510:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829918.914515:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829918/real 1787829918] req@ffff9e6804421680 x1874668790815616/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829918.914521:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790815616/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829918.914527:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829918.914528:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829918.914530:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790815616:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829918.914533:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790815744:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829918.914541:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829918.914546:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829918/real 1787829918] req@ffff9e6804422580 x1874668790815744/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829918.914552:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790815744/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829918.914556:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829918.914557:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829918.914559:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790815744:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829918.914562:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790815872:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829918.914571:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829918.914576:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829918/real 1787829918] req@ffff9e6804420f00 x1874668790815872/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829918.914581:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790815872/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829918.914586:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829918.914587:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829918.914589:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790815872:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829918.914597:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790816000:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829918.914610:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829918.914619:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829918/real 1787829918] req@ffff9e6804423480 x1874668790816000/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829918.914628:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790816000/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829918.914635:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829918.914638:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829918.914641:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790816000:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829919.425812:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829919.617921:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829920.449770:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829920.641911:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829921.473802:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829921.666089:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829922.497824:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829922.690071:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829923.521850:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829923.713862:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829924.033752:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829924.033760:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829924.033764:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829924.033771:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6560 00000100:00080000:2.0:1787829924.033779:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829924.033784:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829924.033814:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790816128/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829924.033834:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829924.033837:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829924.033838:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829924.033840:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6560 00000100:00080000:2.0:1787829924.033843:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829924.033846:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829924.033854:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829924.033862:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790816256/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829924.033865:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790816128:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829924.033875:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829924.033878:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829924.033879:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829924.033881:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6560 00000100:00080000:2.0:1787829924.033883:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829924.033885:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829924.033887:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829924.033894:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790816384/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829924.033899:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829924.033901:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829924.033902:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829924.033904:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6560 00000100:00080000:2.0:1787829924.033907:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829924.033909:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829924.033912:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829924.033919:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790816512/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829924.033927:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829924.033935:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829924.033937:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829924.033939:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829924.033942:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6560 00000100:00080000:2.0:1787829924.033944:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829924.033946:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829924.033948:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829924.033954:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790816640/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829924.033959:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829924.033961:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829924.033962:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829924.033964:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6560 00000100:00080000:2.0:1787829924.033967:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829924.033971:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829924/real 1787829924] req@ffff9e6804420780 x1874668790816128/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829924.033971:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829924.033977:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829924.033987:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790816768/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829924.033993:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790816128/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829924.033994:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829924.034001:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829924.034004:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829924.034012:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6560 00000100:00080000:1.0:1787829924.034013:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829924.034015:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829924.034018:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829924.034023:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790816128:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829924.034025:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829924.034030:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829924.034037:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790816896/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829924.034041:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790816256:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829924.034046:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829924.034049:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829924.034051:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829924.034054:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6560 00000100:00080000:2.0:1787829924.034057:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829924.034063:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829924.034064:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829924.034070:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829924.034077:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829924/real 1787829924] req@ffff9e6804422d00 x1874668790816256/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829924.034086:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790817024/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829924.034090:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790816256/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829924.034097:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829924.034104:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829924.034105:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829924.034106:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829924.034110:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6560 00000100:00080000:1.0:1787829924.034113:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829924.034115:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829924.034118:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790816256:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829924.034121:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829924.034126:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829924.034128:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790816384:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829924.034134:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790817152/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829924.034145:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829924.034152:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829924/real 1787829924] req@ffff9e6804422940 x1874668790816384/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829924.034164:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790816384/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829924.034170:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829924.034172:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829924.034174:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790816384:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829924.034178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790816512:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829924.034187:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829924.034192:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829924/real 1787829924] req@ffff9e68044221c0 x1874668790816512/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829924.034198:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790816512/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829924.034203:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829924.034204:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829924.034206:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790816512:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829924.034208:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790816640:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829924.034219:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829924.034228:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829924/real 1787829924] req@ffff9e6804423840 x1874668790816640/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829924.034363:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790816640/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829924.034371:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829924.034373:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829924.034375:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790816640:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829924.034379:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790816768:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829924.034393:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829924.034399:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829924/real 1787829924] req@ffff9e6804423480 x1874668790816768/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829924.034405:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790816768/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829924.034411:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829924.034412:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829924.034414:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790816768:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829924.034425:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790816896:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829924.034434:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829924.034439:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829924/real 1787829924] req@ffff9e6804420f00 x1874668790816896/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829924.034446:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790816896/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829924.034450:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829924.034451:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829924.034453:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790816896:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829924.034456:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790817024:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829924.034465:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829924.034469:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829924/real 1787829924] req@ffff9e6804422580 x1874668790817024/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829924.034475:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790817024/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829924.034479:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829924.034480:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829924.034489:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790817024:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829924.034492:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790817152:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829924.034502:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829924.034507:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829924/real 1787829924] req@ffff9e6804421680 x1874668790817152/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829924.034513:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790817152/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829924.034517:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829924.034519:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829924.034520:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790817152:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829924.545820:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829924.737842:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829925.569827:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829925.761842:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829926.593758:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:0.0:1787829926.785880:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829927.617761:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829927.809893:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829928.641797:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829928.833888:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829929.153898:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829929.153906:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829929.153909:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829929.153916:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6565 00000100:00080000:2.0:1787829929.153925:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829929.153931:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829929.153965:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790817280/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829929.153991:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829929.153998:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829929.154000:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829929.154004:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6565 00000100:00080000:2.0:1787829929.154009:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829929.154013:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829929.154023:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829929.154032:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790817280:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829929.154039:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790817408/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829929.154047:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829929.154050:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829929.154051:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829929.154053:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6565 00000100:00080000:2.0:1787829929.154056:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829929.154059:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829929.154064:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829929.154075:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790817536/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829929.154085:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829929.154090:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787829929.154091:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829929.154094:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829929.154101:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6565 00000100:00080000:2.0:1787829929.154106:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829929.154109:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829929.154114:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829929.154124:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790817664/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829929.154130:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829929.154133:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829929/real 1787829929] req@ffff9e6804420b40 x1874668790817280/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829929.154133:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829929.154136:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829929.154143:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6565 00000100:00080000:2.0:1787829929.154148:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829929.154152:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829929.154155:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829929.154159:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790817280/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829929.154164:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790817792/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829929.154175:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829929.154176:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829929.154181:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829929.154181:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829929.154185:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829929.154189:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790817280:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829929.154192:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6565 00000100:00080000:2.0:1787829929.154211:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829929.154215:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829929.154218:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829929.154226:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790817920/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829929.154229:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790817408:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829929.154241:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829929.154244:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829929.154245:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829929.154249:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6565 00000100:00080000:2.0:1787829929.154252:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829929.154255:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1787829929.154258:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:2.0:1787829929.154261:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829929.154272:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790818048/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829929.154273:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829929/real 1787829929] req@ffff9e68044230c0 x1874668790817408/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829929.154281:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829929.154283:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829929.154284:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829929.154287:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6565 00000100:00080000:2.0:1787829929.154291:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829929.154293:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829929.154296:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829929.154306:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790818176/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829929.154313:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790817408/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829929.154313:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829929.154318:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829929.154321:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829929.154329:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6565 00000100:00080000:2.0:1787829929.154332:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829929.154333:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829929.154336:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829929.154341:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787829929.154343:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829929.154348:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790818304/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829929.154351:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790817408:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829929.154366:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790817536:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829929.154391:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829929.154400:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829929/real 1787829929] req@ffff9e68044212c0 x1874668790817536/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829929.154411:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790817536/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829929.154416:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829929.154418:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829929.154420:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790817536:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829929.154423:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790817664:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829929.154434:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829929.154439:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829929/real 1787829929] req@ffff9e6804421e00 x1874668790817664/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829929.154445:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790817664/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829929.154450:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829929.154451:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829929.154453:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790817664:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829929.154456:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790817792:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829929.154465:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829929.154469:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829929/real 1787829929] req@ffff9e6804421680 x1874668790817792/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829929.154475:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790817792/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829929.154483:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829929.154485:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829929.154487:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790817792:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829929.154498:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790817920:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829929.154507:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829929.154512:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829929/real 1787829929] req@ffff9e6804422580 x1874668790817920/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829929.154518:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790817920/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829929.154525:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829929.154528:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829929.154546:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790817920:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829929.154552:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790818048:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829929.154569:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829929.154578:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829929/real 1787829929] req@ffff9e6804420f00 x1874668790818048/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829929.154588:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790818048/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829929.154595:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829929.154597:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829929.154600:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790818048:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829929.154605:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790818176:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829929.154621:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829929.154630:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829929/real 1787829929] req@ffff9e6804423480 x1874668790818176/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829929.154638:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790818176/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829929.154646:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829929.154648:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829929.154651:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790818176:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829929.154655:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790818304:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829929.154669:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829929.154676:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829929/real 1787829929] req@ffff9e6804423840 x1874668790818304/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829929.154685:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790818304/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829929.154691:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829929.154717:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829929.154721:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790818304:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829929.665814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829929.857912:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829930.689800:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829930.881869:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829931.713806:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829931.905912:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829932.737781:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829932.929879:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829933.761835:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829933.956260:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829934.273810:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829934.273820:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829934.273825:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829934.273836:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6570 00000100:00080000:2.0:1787829934.273846:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829934.273854:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829934.273891:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790818432/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829934.273920:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829934.273923:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829934.273929:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829934.273933:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6570 00000100:00080000:2.0:1787829934.273937:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829934.273941:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829934.273951:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829934.273966:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790818560/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829934.273977:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829934.273983:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790818432:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829934.273983:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829934.273986:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829934.273996:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6570 00000100:00080000:2.0:1787829934.274000:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829934.274004:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829934.274009:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829934.274019:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790818688/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829934.274030:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829934.274034:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829934.274035:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829934.274040:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6570 00000100:00080000:2.0:1787829934.274043:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829934.274048:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1787829934.274054:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:2.0:1787829934.274056:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829934.274069:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790818816/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829934.274079:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829934.274083:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829934.274084:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829934.274089:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6570 00000100:00080000:2.0:1787829934.274093:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829934.274098:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829934/real 1787829934] req@ffff9e68044221c0 x1874668790818432/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829934.274099:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829934.274105:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829934.274121:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790818944/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829934.274127:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790818432/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829934.274136:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829934.274139:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829934.274141:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829934.274146:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6570 00000100:00080000:2.0:1787829934.274153:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829934.274171:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787829934.274173:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787829934.274177:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829934.274182:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829934.274188:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790818432:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829934.274193:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790819072/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829934.274205:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829934.274209:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829934.274211:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829934.274214:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6570 00000100:00080000:2.0:1787829934.274218:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829934.274222:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829934.274227:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829934.274230:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790818560:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829934.274239:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790819200/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829934.274250:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829934.274253:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829934.274255:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829934.274261:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6570 00000800:00000100:1.0:1787829934.274263:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829934.274266:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829934.274272:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829934.274277:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829934.274283:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829934/real 1787829934] req@ffff9e6804422940 x1874668790818560/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829934.274290:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790819328/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829934.274304:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829934.274306:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790818560/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829934.274309:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829934.274313:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829934.274322:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6570 00000100:00080000:1.0:1787829934.274326:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829934.274328:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829934.274335:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787829934.274337:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:2.0:1787829934.274343:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829934.274344:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790818560:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829934.274356:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790818688:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829934.274359:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790819456/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829934.274385:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829934.274400:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829934/real 1787829934] req@ffff9e6804422d00 x1874668790818688/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829934.274430:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790818688/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829934.274438:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829934.274441:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829934.274445:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790818688:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829934.274451:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790818816:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829934.274478:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829934.274490:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829934/real 1787829934] req@ffff9e6804420780 x1874668790818816/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829934.274501:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790818816/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829934.274509:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829934.274522:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829934.274525:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790818816:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829934.274531:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790818944:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829934.274550:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829934.274560:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829934/real 1787829934] req@ffff9e6804423840 x1874668790818944/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829934.274571:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790818944/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829934.274584:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829934.274587:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829934.274591:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790818944:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829934.274596:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790819072:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829934.274615:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829934.274638:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829934/real 1787829934] req@ffff9e6804423480 x1874668790819072/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829934.274649:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790819072/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829934.274658:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829934.274661:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829934.274664:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790819072:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829934.274682:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790819200:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829934.274733:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829934.274749:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829934/real 1787829934] req@ffff9e6804420f00 x1874668790819200/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829934.274760:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790819200/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829934.274767:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829934.274770:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829934.274774:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790819200:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829934.274780:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790819328:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829934.274799:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829934.274809:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829934/real 1787829934] req@ffff9e6804422580 x1874668790819328/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829934.274821:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790819328/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829934.274828:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829934.274831:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829934.274834:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790819328:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829934.274839:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790819456:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829934.274858:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829934.274869:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829934/real 1787829934] req@ffff9e6804421680 x1874668790819456/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829934.274880:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790819456/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829934.274888:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829934.274891:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829934.274894:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790819456:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829934.785842:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829934.980334:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829935.809827:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829936.001876:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829936.833824:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829937.025883:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829937.857798:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829938.049911:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829938.881806:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829939.073897:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829939.393787:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829939.393798:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829939.393803:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829939.393812:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6576 00000100:00080000:2.0:1787829939.393822:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829939.393830:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829939.393880:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790819584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829939.393913:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829939.393917:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829939.393919:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829939.393923:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6576 00000100:00080000:2.0:1787829939.393927:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829939.393930:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829939.393940:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829939.393954:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790819712/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829939.393964:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790819584:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829939.393967:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829939.393972:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829939.393974:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829939.393979:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6576 00000100:00080000:2.0:1787829939.393984:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829939.393988:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829939.393993:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829939.394005:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790819840/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829939.394018:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829939.394026:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829939.394030:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829939.394032:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829939.394037:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6576 00000100:00080000:2.0:1787829939.394042:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829939.394047:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829939.394052:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829939.394056:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829939/real 1787829939] req@ffff9e6804421e00 x1874668790819584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829939.394067:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790819968/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829939.394077:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790819584/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829939.394080:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829939.394089:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829939.394093:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829939.394099:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829939.394100:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6576 00000100:00080000:1.0:1787829939.394109:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829939.394109:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829939.394118:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790819584:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829939.394119:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829939.394139:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829939.394153:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790820096/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829939.394160:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790819712:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829939.394168:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829939.394172:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829939.394174:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829939.394179:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6576 00000100:00080000:2.0:1787829939.394185:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829939.394190:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829939.394194:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829939.394201:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829939.394203:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829939/real 1787829939] req@ffff9e68044212c0 x1874668790819712/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829939.394219:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790819712/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829939.394219:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790820224/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829939.394235:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829939.394236:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829939.394242:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829939.394242:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829939.394247:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829939.394248:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790819712:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829939.394257:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6576 00000100:00100000:1.0:1787829939.394258:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790819840:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829939.394265:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829939.394272:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829939.394277:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829939.394279:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829939.394289:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829939/real 1787829939] req@ffff9e68044230c0 x1874668790819840/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829939.394295:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790820352/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829939.394316:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790819840/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829939.394318:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829939.394322:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829939.394325:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829939.394334:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6576 00000100:00080000:1.0:1787829939.394339:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829939.394341:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829939.394349:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829939.394350:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829939.394359:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829939.394360:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790819840:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829939.394371:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790819968:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829939.394376:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790820480/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829939.394393:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829939.394394:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829939.394399:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829939.394401:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829939.394403:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829939/real 1787829939] req@ffff9e6804420b40 x1874668790819968/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829939.394411:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790819968/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829939.394426:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6576 00000100:00080000:1.0:1787829939.394427:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829939.394433:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829939.394435:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829939.394441:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829939.394443:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790819968:192.168.203.109@tcp:8:kworker.0 00000008:00100000:2.0:1787829939.394449:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829939.394463:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790820096:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829939.394470:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790820608/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829939.394497:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829939.394510:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829939/real 1787829939] req@ffff9e6804421680 x1874668790820096/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829939.394521:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790820096/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829939.394531:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829939.394533:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829939.394535:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790820096:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829939.394539:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790820224:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829939.394552:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829939.394558:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829939/real 1787829939] req@ffff9e6804422580 x1874668790820224/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829939.394564:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790820224/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829939.394569:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829939.394571:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829939.394572:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790820224:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829939.394576:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790820352:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829939.394584:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829939.394589:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829939/real 1787829939] req@ffff9e6804420f00 x1874668790820352/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829939.394595:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790820352/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829939.394599:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829939.394600:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829939.394609:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790820352:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829939.394613:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790820480:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829939.394623:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829939.394628:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829939/real 1787829939] req@ffff9e6804423480 x1874668790820480/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829939.394635:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790820480/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829939.394641:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829939.394644:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829939.394647:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790820480:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829939.394661:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790820608:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829939.394679:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829939.394689:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829939/real 1787829939] req@ffff9e6804423840 x1874668790820608/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829939.394716:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790820608/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829939.394720:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829939.394722:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829939.394724:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790820608:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829939.905779:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829940.097870:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829940.929816:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829941.121890:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829941.953796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829942.145899:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829942.977822:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829943.169922:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829944.001820:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829944.193896:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829944.513981:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829944.513991:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829944.513994:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829944.514002:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6581 00000100:00080000:0.0:1787829944.514012:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829944.514019:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829944.514057:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790820736/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829944.514086:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829944.514100:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829944.514103:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829944.514107:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6581 00000100:00080000:0.0:1787829944.514111:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829944.514115:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829944.514126:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829944.514139:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790820864/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829944.514149:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829944.514152:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829944.514153:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829944.514158:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6581 00000100:00080000:0.0:1787829944.514161:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829944.514164:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829944.514169:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829944.514172:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790820736:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787829944.514180:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790820992/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829944.514190:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829944.514193:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829944.514195:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829944.514210:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6581 00000100:00080000:0.0:1787829944.514223:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829944.514227:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1787829944.514232:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787829944.514233:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829944.514246:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790821120/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829944.514256:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829944.514259:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829944.514261:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829944.514265:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6581 00000100:00080000:0.0:1787829944.514269:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829944.514273:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829944/real 1787829944] req@ffff9e6803e9a580 x1874668790820736/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829944.514276:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829944.514283:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829944.514298:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790820736/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829944.514306:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790821248/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829944.514323:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829944.514323:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829944.514330:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829944.514332:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829944.514333:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829944.514340:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790820736:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829944.514341:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6581 00000100:00080000:0.0:1787829944.514347:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829944.514353:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790820864:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829944.514354:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829944.514361:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829944.514373:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790821376/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829944.514382:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829944.514385:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829944.514388:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829944.514391:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829944.514396:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6581 00000100:00080000:0.0:1787829944.514401:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829944.514401:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829944/real 1787829944] req@ffff9e6803e99680 x1874668790820864/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829944.514409:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829944.514417:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829944.514424:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790820864/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829944.514430:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790821504/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829944.514437:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829944.514445:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829944.514448:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829944.514450:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829944.514457:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6581 00000100:00080000:1.0:1787829944.514457:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829944.514463:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829944.514465:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790820864:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829944.514469:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829944.514474:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829944.514484:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790821632/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829944.514492:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790820992:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829944.514495:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829944.514500:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829944.514502:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829944.514506:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6581 00000100:00080000:0.0:1787829944.514512:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787829944.514516:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829944.514517:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829944.514523:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829944.514531:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829944/real 1787829944] req@ffff9e6803e99e00 x1874668790820992/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829944.514535:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790821760/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829944.514567:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790820992/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829944.514575:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829944.514578:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829944.514588:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790820992:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829944.514594:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790821120:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829944.514612:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829944.514621:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829944/real 1787829944] req@ffff9e6803e9b480 x1874668790821120/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829944.514631:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790821120/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829944.514639:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829944.514641:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829944.514644:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790821120:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829944.514649:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790821248:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829944.514668:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829944.514677:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829944/real 1787829944] req@ffff9e6803e9b840 x1874668790821248/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829944.514687:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790821248/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829944.514722:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829944.514725:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829944.514728:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790821248:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829944.514733:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790821376:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829944.514750:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829944.514759:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829944/real 1787829944] req@ffff9e6803e9a1c0 x1874668790821376/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829944.514769:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790821376/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829944.514777:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829944.514780:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829944.514783:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790821376:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829944.514789:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790821504:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829944.514802:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829944.514811:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829944/real 1787829944] req@ffff9e6803e98780 x1874668790821504/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829944.514820:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790821504/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829944.514828:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829944.514830:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829944.514842:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790821504:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829944.514859:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790821632:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829944.514876:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829944.514885:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829944/real 1787829944] req@ffff9e6803e98b40 x1874668790821632/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829944.514895:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790821632/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829944.514903:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829944.514905:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829944.514908:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790821632:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829944.514913:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790821760:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829944.514927:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829944.514935:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829944/real 1787829944] req@ffff9e6803e992c0 x1874668790821760/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829944.514945:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790821760/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829944.514952:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829944.514955:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829944.514958:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790821760:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829945.025816:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829945.217914:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829946.049779:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829946.241827:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829947.073808:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829947.265924:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829948.098248:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829948.289846:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829949.121804:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829949.313875:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829949.633873:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829949.633885:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829949.633891:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829949.633905:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6586 00000100:00080000:0.0:1787829949.633920:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829949.633931:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829949.633981:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790821888/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829949.634016:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829949.634020:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829949.634023:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829949.634029:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6586 00000100:00080000:0.0:1787829949.634037:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829949.634044:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829949.634066:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790821888:192.168.203.109@tcp:250:kworker.0 00000008:00100000:0.0:1787829949.634075:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829949.634103:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790822016/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829949.634118:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829949.634123:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829949.634126:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829949.634131:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6586 00000100:00080000:0.0:1787829949.634141:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829949.634147:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787829949.634151:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787829949.634158:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829949.634174:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790822144/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829949.634186:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829949.634190:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829949.634194:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829949.634195:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829949/real 1787829949] req@ffff9e6803e98f00 x1874668790821888/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829949.634205:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6586 00000100:00080000:0.0:1787829949.634213:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829949.634219:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829949.634222:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790821888/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829949.634230:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787829949.634234:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829949.634240:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829949.634249:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790821888:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787829949.634250:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790822272/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829949.634264:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829949.634268:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829949.634270:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829949.634276:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6586 00000100:00100000:2.0:1787829949.634281:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790822016:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829949.634283:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829949.634292:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829949.634298:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829949.634310:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790822400/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787829949.634319:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829949.634328:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829949.634332:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829949.634334:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829949.634339:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6586 00000100:00100000:2.0:1787829949.634341:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829949/real 1787829949] req@ffff9e6803e9ad00 x1874668790822016/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829949.634343:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829949.634353:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829949.634360:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790822016/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829949.634362:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829949.634375:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829949.634378:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787829949.634384:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790822528/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829949.634384:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790822016:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829949.634396:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790822144:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829949.634406:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829949.634411:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829949.634414:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829949.634421:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6586 00000800:00000100:2.0:1787829949.634422:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829949.634430:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829949.634436:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829949.634443:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829949.634447:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829949/real 1787829949] req@ffff9e6803e9bc00 x1874668790822144/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829949.634459:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790822656/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829949.634464:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790822144/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829949.634487:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829949.634488:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829949.634494:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829949.634496:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829949.634501:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829949.634506:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790822144:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829949.634507:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6586 00000100:00080000:0.0:1787829949.634516:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829949.634517:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790822272:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787829949.634526:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829949.634532:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787829949.634539:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829949.634549:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790822784/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829949.634553:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829949/real 1787829949] req@ffff9e6803e9a940 x1874668790822272/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829949.634582:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829949.634588:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790822272/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829949.634589:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829949.634593:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829949.634602:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6586 00000100:00080000:2.0:1787829949.634609:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829949.634611:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829949.634620:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829949.634622:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829949.634630:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790822272:192.168.203.109@tcp:8:kworker.0 00000008:00100000:0.0:1787829949.634631:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829949.634652:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790822912/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829949.634658:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790822400:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829949.634686:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829949.634725:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829949/real 1787829949] req@ffff9e6803e98f00 x1874668790822400/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829949.634735:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790822400/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829949.634743:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829949.634745:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829949.634748:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790822400:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829949.634752:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790822528:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829949.634768:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829949.634780:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829949/real 1787829949] req@ffff9e6803e992c0 x1874668790822528/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829949.634796:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790822528/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829949.634806:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829949.634809:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829949.634812:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790822528:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829949.634816:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790822656:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829949.634827:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829949.634833:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829949/real 1787829949] req@ffff9e6803e98b40 x1874668790822656/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829949.634841:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790822656/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829949.634846:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829949.634848:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829949.634850:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790822656:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829949.634853:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790822784:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829949.634863:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829949.634869:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829949/real 1787829949] req@ffff9e6803e98780 x1874668790822784/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829949.634881:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790822784/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829949.634895:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829949.634899:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829949.634903:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790822784:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829949.634920:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790822912:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829949.634943:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829949.634951:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829949/real 1787829949] req@ffff9e6803e9a1c0 x1874668790822912/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829949.634959:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790822912/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829949.634965:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829949.634967:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829949.634969:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790822912:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829950.145776:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829950.337904:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829951.169905:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829951.361890:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829952.193785:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829952.385899:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829953.217847:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829953.409883:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829954.241790:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829954.433915:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829954.753817:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829954.753827:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829954.753832:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829954.753841:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6591 00000100:00080000:3.0:1787829954.753851:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829954.753856:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829954.753886:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790823040/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829954.753905:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829954.753908:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829954.753910:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829954.753914:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6591 00000100:00080000:3.0:1787829954.753918:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829954.753922:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829954.753932:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829954.753935:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790823040:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829954.753944:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790823168/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829954.753952:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829954.753954:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829954.753956:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829954.753958:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6591 00000100:00080000:3.0:1787829954.753961:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829954.753963:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829954.753966:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829954.753973:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790823296/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829954.753979:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829954.753982:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829954.753983:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829954.753985:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6591 00000800:00000100:2.0:1787829954.753989:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829954.753991:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829954.753996:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829954.754000:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829954.754005:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790823424/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829954.754011:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829954.754013:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829954.754015:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829954.754029:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6591 00000100:00080000:3.0:1787829954.754032:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829954.754035:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829954.754038:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829954.754047:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829954/real 1787829954] req@ffff9e680a2d43c0 x1874668790823040/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829954.754048:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790823552/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829954.754072:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829954.754075:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829954.754076:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829954.754080:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6591 00000100:00100000:2.0:1787829954.754081:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790823040/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829954.754082:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829954.754087:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829954.754092:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787829954.754103:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829954.754104:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790823680/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829954.754112:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829954.754121:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790823040:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829954.754122:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829954.754130:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829954.754132:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829954.754137:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6591 00000100:00080000:3.0:1787829954.754141:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829954.754144:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829954.754148:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829954.754152:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790823168:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829954.754158:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790823808/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829954.754166:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829954.754168:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829954.754170:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829954.754173:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6591 00000100:00080000:3.0:1787829954.754176:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829954.754179:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:2.0:1787829954.754183:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:3.0:1787829954.754185:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829954.754198:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790823936/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829954.754202:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829954/real 1787829954] req@ffff9e680a2d4f00 x1874668790823168/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829954.754212:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829954.754220:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829954.754222:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829954.754227:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6591 00000100:00100000:2.0:1787829954.754228:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790823168/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829954.754229:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829954.754233:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829954.754236:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829954.754244:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790824064/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829954.754250:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829954.754257:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829954.754261:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790823168:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829954.754269:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790823296:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829954.754291:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829954.754301:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829954/real 1787829954] req@ffff9e680a2d5a40 x1874668790823296/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829954.754309:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790823296/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829954.754314:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829954.754316:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829954.754318:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790823296:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829954.754322:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790823424:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829954.754335:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829954.754340:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829954/real 1787829954] req@ffff9e680a2d6d00 x1874668790823424/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829954.754347:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790823424/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829954.754352:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829954.754353:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829954.754355:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790823424:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829954.754358:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790823552:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829954.754367:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829954.754372:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829954/real 1787829954] req@ffff9e680a2d7c00 x1874668790823552/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829954.754378:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790823552/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829954.754382:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829954.754384:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829954.754385:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790823552:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829954.754388:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790823680:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829954.754399:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829954.754404:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829954/real 1787829954] req@ffff9e680a2d5e00 x1874668790823680/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829954.754411:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790823680/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829954.754416:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829954.754418:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829954.754419:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790823680:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829954.754432:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790823808:192.168.203.109@tcp:38:kworker.0 00000800:00000100:2.0:1787829954.754442:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829954.754447:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829954/real 1787829954] req@ffff9e680a2d7480 x1874668790823808/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829954.754453:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790823808/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829954.754458:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829954.754465:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829954.754467:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790823808:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829954.754471:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790823936:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829954.754480:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829954.754486:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829954/real 1787829954] req@ffff9e680a2d52c0 x1874668790823936/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829954.754492:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790823936/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829954.754500:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829954.754502:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829954.754504:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790823936:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829954.754506:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790824064:192.168.203.109@tcp:8:kworker.0 00000800:00000100:2.0:1787829954.754520:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787829954.754526:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829954/real 1787829954] req@ffff9e680a2d61c0 x1874668790824064/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829954.754532:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790824064/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829954.754536:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829954.754538:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:2.0:1787829954.754539:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790824064:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829955.265824:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829955.457896:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829956.289814:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829956.481924:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829957.313820:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829957.505920:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829958.337829:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829958.529934:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829959.361796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829959.554090:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829959.873842:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829959.873851:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829959.873855:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829959.873865:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6596 00000100:00080000:2.0:1787829959.873876:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829959.873883:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787829959.873918:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790824192/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829959.873951:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829959.873955:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829959.873957:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829959.873960:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6596 00000100:00080000:2.0:1787829959.873964:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829959.873967:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829959.873976:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829959.874000:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790824320/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829959.874010:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829959.874014:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829959.874017:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829959.874018:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790824192:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829959.874027:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6596 00000100:00080000:2.0:1787829959.874032:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829959.874036:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829959.874042:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829959.874054:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790824448/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829959.874062:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829959.874066:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787829959.874077:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829959.874124:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829959/real 1787829959] req@ffff9e6804420780 x1874668790824192/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829959.874182:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790824192/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829959.874193:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829959.874197:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829959.874201:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6596 00000100:00080000:1.0:1787829959.874206:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829959.874209:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829959.874212:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790824192:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787829959.874217:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829959.874225:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829959.874240:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790824320:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787829959.874241:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790824576/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829959.874256:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829959.874259:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829959.874261:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829959.874265:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6596 00000800:00000100:1.0:1787829959.874269:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787829959.874270:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829959.874277:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829959.874283:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829959.874287:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829959/real 1787829959] req@ffff9e6804422d00 x1874668790824320/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829959.874298:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790824704/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829959.874302:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790824320/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829959.874320:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829959.874321:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829959.874327:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829959.874328:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787829959.874332:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829959.874335:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790824320:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829959.874340:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6596 00000100:00080000:2.0:1787829959.874347:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829959.874352:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790824448:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787829959.874355:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829959.874363:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829959.874374:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790824832/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829959.874383:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829959.874386:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787829959.874388:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829959.874392:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6596 00000100:00080000:2.0:1787829959.874395:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829959.874398:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829959.874403:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829959.874412:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790824960/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829959.874420:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829959.874423:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829959.874425:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829959.874428:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6596 00000100:00080000:2.0:1787829959.874431:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829959.874435:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829959.874439:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829959.874449:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790825088/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829959.874457:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829959.874460:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829959.874462:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829959.874466:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6596 00000100:00080000:2.0:1787829959.874470:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829959.874473:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787829959.874478:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787829959.874487:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790825216/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829959.875417:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829959.875441:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829959/real 1787829959] req@ffff9e6804422940 x1874668790824448/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829959.875455:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790824448/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829959.875463:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829959.875467:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829959.875471:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790824448:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829959.875488:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790824576:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829959.875506:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829959.875515:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829959/real 1787829959] req@ffff9e68044221c0 x1874668790824576/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829959.875526:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790824576/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829959.875534:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829959.875537:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829959.875540:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790824576:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829959.875546:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790824704:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829959.875560:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829959.875568:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829959/real 1787829959] req@ffff9e6804420780 x1874668790824704/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829959.875579:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790824704/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829959.875587:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829959.875590:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829959.875593:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790824704:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829959.875598:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790824832:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829959.875612:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829959.875620:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829959/real 1787829959] req@ffff9e6804423840 x1874668790824832/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829959.875631:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790824832/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829959.875639:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829959.875642:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829959.875645:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790824832:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829959.875650:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790824960:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829959.875664:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829959.875672:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829959/real 1787829959] req@ffff9e6804423480 x1874668790824960/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829959.875682:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790824960/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829959.875709:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829959.875712:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829959.875715:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790824960:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829959.875721:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790825088:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829959.875737:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829959.875745:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829959/real 1787829959] req@ffff9e6804420f00 x1874668790825088/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829959.875756:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790825088/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829959.875763:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829959.875766:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829959.875769:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790825088:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829959.875773:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790825216:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829959.875786:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829959.875794:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829959/real 1787829959] req@ffff9e6804422580 x1874668790825216/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829959.875804:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790825216/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829959.875812:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829959.875814:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829959.875818:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790825216:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829960.385828:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829960.577903:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829961.409818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829961.602641:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829962.435724:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829962.625948:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829963.457820:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829963.650247:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829964.481818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829964.673890:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829964.993857:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829964.993868:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829964.993872:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829964.993882:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6601 00000100:00080000:3.0:1787829964.993892:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829964.993899:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829964.993933:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790825344/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829964.993968:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829964.993972:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829964.993977:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829964.993981:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6601 00000100:00080000:3.0:1787829964.993984:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829964.993988:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829964.993998:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829964.994011:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790825472/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829964.994019:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790825344:192.168.203.109@tcp:250:kworker.0 00000100:00080000:3.0:1787829964.994024:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829964.994029:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829964.994031:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829964.994035:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6601 00000100:00080000:3.0:1787829964.994039:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829964.994042:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829964.994046:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829964.994058:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790825600/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829964.994067:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829964.994070:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829964.994072:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829964.994076:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6601 00000100:00080000:3.0:1787829964.994079:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829964.994083:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829964.994087:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829964.994097:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790825728/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829964.994106:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829964.994110:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829964.994112:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829964.994116:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6601 00000100:00080000:3.0:1787829964.994119:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829964.994122:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829964.994127:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829964.994133:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790825472:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829964.994139:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790825856/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829964.994149:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829964.994152:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829964.994154:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829964.994157:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6601 00000100:00080000:3.0:1787829964.994160:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829964.994165:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829964.994170:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829964.994175:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790825600:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829964.994181:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790825984/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829964.994190:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829964.994194:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829964.994196:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829964.994200:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790825728:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829964.994201:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6601 00000100:00080000:3.0:1787829964.994208:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829964.994213:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829964.994219:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829964.994230:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790826112/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829964.994239:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829964.994242:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790825856:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787829964.994243:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829964.994247:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829964.994254:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6601 00000100:00080000:3.0:1787829964.994257:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829964.994262:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829964.994266:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829964.994278:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790826240/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829964.994283:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790825984:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787829964.994292:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829964.994296:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829964.994298:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829964.994302:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6601 00000100:00080000:3.0:1787829964.994306:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829964.994310:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829964.994314:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829964.994321:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790826112:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829964.994325:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790826368/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829964.994359:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790826240:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829964.994380:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790826368:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829965.505813:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829965.697892:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829966.530086:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829966.722581:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829967.553820:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829967.745868:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787829968.068766:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:2.0:1787829968.068830:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787829968.068851:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787829968.068930:0:2389:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:2.0:1787829968.068945:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787829968.068954:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787829968.068961:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787829968.068970:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787829968.068977:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787829968.068985:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:2.0:1787829968.068993:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:1.0:1787829968.069083:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829964/real 1787829968] req@ffff9e680a2d4f00 x1874668790826368/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829968.069110:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790826368/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829968.069122:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829968.069127:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829968.069132:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790826368:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829968.069140:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829964/real 1787829968] req@ffff9e680a2d6d00 x1874668790826112/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829968.069153:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790826112/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829968.069160:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829968.069162:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829968.069165:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790826112:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829968.069170:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829964/real 1787829968] req@ffff9e680a2d5a40 x1874668790826240/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829968.069179:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790826240/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829968.069186:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829968.069190:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829968.069194:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790826240:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829968.069200:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829964/real 1787829968] req@ffff9e680a2d5e00 x1874668790825856/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829968.069380:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790825856/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829968.069390:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829968.069393:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829968.069407:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790825856:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829968.069416:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829964/real 1787829968] req@ffff9e680a2d7c00 x1874668790825984/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829968.069429:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790825984/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829968.069450:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829968.069454:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829968.069457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790825984:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829968.069465:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829964/real 1787829968] req@ffff9e680a2d61c0 x1874668790825472/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829968.069477:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790825472/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829968.069485:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829968.069489:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829968.069492:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790825472:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829968.069498:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829964/real 1787829968] req@ffff9e680a2d52c0 x1874668790825600/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829968.069505:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790825600/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829968.069510:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829968.069513:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829968.069515:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790825600:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829968.069519:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829964/real 1787829968] req@ffff9e680a2d7480 x1874668790825728/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829968.069526:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790825728/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829968.069530:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829968.069531:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829968.069533:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790825728:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829968.069537:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829964/real 1787829968] req@ffff9e680a2d4b40 x1874668790825344/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829968.069551:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790825344/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829968.069558:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829968.069561:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829968.069564:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790825344:192.168.203.109@tcp:250:kworker.0 00000100:00100000:2.0:1787829968.577830:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829968.769972:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829969.601787:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829969.794424:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829970.113805:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829970.113812:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829970.113816:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829970.113824:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6606 00000100:00080000:3.0:1787829970.113832:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829970.113838:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829970.113864:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790826496/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829970.113889:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829970.113891:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829970.113892:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829970.113895:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6606 00000100:00080000:3.0:1787829970.113897:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829970.113899:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829970.113906:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829970.113924:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790826624/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829970.113930:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829970.113932:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829970.113933:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829970.113935:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6606 00000100:00080000:3.0:1787829970.113938:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829970.113941:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829970.113945:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829970.113954:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790826752/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829970.113963:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829970.113966:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829970.113968:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829970.113971:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6606 00000100:00080000:3.0:1787829970.113974:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829970.113977:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829970.113981:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829970.113991:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790826880/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829970.113996:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829970.113998:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829970.113999:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829970.114001:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6606 00000100:00080000:3.0:1787829970.114003:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829970.114005:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829970.114008:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829970.114016:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790827008/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829970.114024:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829970.114026:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829970.114029:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829970.114032:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6606 00000100:00080000:3.0:1787829970.114035:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829970.114039:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829970.114043:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829970.114052:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790827136/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829970.114060:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829970.114063:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829970.114065:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829970.114069:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6606 00000100:00080000:3.0:1787829970.114072:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829970.114075:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829970.114080:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829970.114090:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790827264/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829970.114098:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829970.114101:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829970.114103:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829970.114106:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6606 00000100:00080000:3.0:1787829970.114109:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829970.114112:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829970.114116:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829970.114126:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790827392/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829970.114133:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829970.114136:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829970.114138:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829970.114141:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6606 00000100:00080000:3.0:1787829970.114144:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829970.114147:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829970.114151:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829970.114160:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790827520/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829970.114839:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790826496:192.168.203.109@tcp:250:kworker.0 00000800:00000100:1.0:1787829970.114903:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829970.114952:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829970/real 1787829970] req@ffff9e680a2d43c0 x1874668790826496/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829970.114967:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790826496/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829970.114981:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829970.114984:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829970.114988:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790826496:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787829970.114995:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790826624:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829970.115016:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829970.115029:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829970/real 1787829970] req@ffff9e680a2d4000 x1874668790826624/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829970.115040:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790826624/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829970.115049:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829970.115052:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829970.115055:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790826624:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829970.115061:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790826752:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829970.115079:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829970.115089:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829970/real 1787829970] req@ffff9e680a2d6580 x1874668790826752/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829970.115098:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790826752/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829970.115105:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829970.115108:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829970.115110:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790826752:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829970.115116:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790826880:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829970.115133:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829970.115143:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829970/real 1787829970] req@ffff9e680a2d70c0 x1874668790826880/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829970.115153:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790826880/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829970.115160:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829970.115163:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829970.115166:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790826880:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829970.115172:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790827008:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829970.115189:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829970.115198:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829970/real 1787829970] req@ffff9e680a2d4780 x1874668790827008/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829970.115207:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790827008/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829970.115214:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829970.115216:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829970.115219:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790827008:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829970.115225:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790827136:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829970.115241:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829970.115251:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829970/real 1787829970] req@ffff9e680a2d4b40 x1874668790827136/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829970.115262:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790827136/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829970.115270:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829970.115273:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829970.115340:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790827136:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829970.115349:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790827264:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829970.115371:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829970.115382:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829970/real 1787829970] req@ffff9e680a2d7480 x1874668790827264/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829970.115393:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790827264/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829970.115400:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829970.115403:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829970.115406:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790827264:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829970.115412:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790827392:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829970.116041:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829970.116055:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829970/real 1787829970] req@ffff9e680a2d52c0 x1874668790827392/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829970.116066:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790827392/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829970.116079:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829970.116082:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829970.116085:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790827392:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829970.116091:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790827520:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829970.116108:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829970.116118:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829970/real 1787829970] req@ffff9e680a2d61c0 x1874668790827520/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829970.116128:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790827520/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829970.116135:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829970.116138:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829970.116140:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790827520:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829970.626983:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829970.817897:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829971.649783:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829971.842044:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829972.673821:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829972.865848:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829973.697835:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829973.890942:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829974.721823:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829974.913927:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829975.233934:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829975.233941:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829975.233945:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829975.233952:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6611 00000100:00080000:3.0:1787829975.233961:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829975.233966:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829975.233995:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790827648/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829975.234018:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829975.234021:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829975.234022:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829975.234024:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6611 00000100:00080000:3.0:1787829975.234027:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829975.234029:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829975.234036:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829975.234045:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790827776/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829975.234051:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829975.234053:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829975.234055:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829975.234057:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6611 00000100:00080000:3.0:1787829975.234058:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829975.234062:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829975.234064:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790827648:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787829975.234066:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829975.234075:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790827904/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829975.234082:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829975.234084:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829975.234085:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829975.234087:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6611 00000100:00080000:3.0:1787829975.234089:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829975.234091:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829975.234094:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829975.234101:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790828032/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829975.234111:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829975.234114:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829975.234115:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787829975.234119:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787829975.234119:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6611 00000100:00080000:3.0:1787829975.234125:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829975.234127:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829975.234130:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829975.234138:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790828160/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829975.234143:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829975.234145:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829975.234146:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829975.234148:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6611 00000100:00080000:3.0:1787829975.234151:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829975.234153:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829975.234156:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829975.234164:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829975/real 1787829975] req@ffff9e680a2d7c00 x1874668790827648/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829975.234165:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790828288/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829975.234190:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829975.234192:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829975.234193:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829975.234196:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6611 00000100:00080000:3.0:1787829975.234199:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829975.234200:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790827648/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829975.234204:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829975.234207:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829975.234213:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790828416/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829975.234234:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829975.234237:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829975.234238:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829975.234239:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829975.234242:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6611 00000100:00080000:3.0:1787829975.234245:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829975.234246:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:3.0:1787829975.234250:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829975.234253:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790827648:192.168.203.109@tcp:250:kworker.0 00000008:00100000:3.0:1787829975.234255:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829975.234264:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790827776:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829975.234266:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790828544/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829975.234272:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829975.234274:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829975.234276:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829975.234278:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6611 00000100:00080000:3.0:1787829975.234281:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829975.234283:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829975.234286:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829975.234293:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829975.234298:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790828672/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829975.234310:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829975/real 1787829975] req@ffff9e680a2d5e00 x1874668790827776/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829975.234326:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790827776/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829975.234332:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829975.234334:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829975.234336:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790827776:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829975.234355:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790827904:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829975.234369:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829975.234375:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829975/real 1787829975] req@ffff9e680a2d5a40 x1874668790827904/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829975.234381:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790827904/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829975.234386:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829975.234388:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829975.234390:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790827904:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829975.234393:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790828032:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829975.234403:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829975.234408:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829975/real 1787829975] req@ffff9e680a2d6d00 x1874668790828032/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829975.234414:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790828032/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829975.234418:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829975.234420:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829975.234421:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790828032:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829975.234424:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790828160:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829975.234433:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829975.234437:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829975/real 1787829975] req@ffff9e680a2d4f00 x1874668790828160/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829975.234443:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790828160/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829975.234448:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829975.234449:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829975.234451:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790828160:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829975.234454:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790828288:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829975.234462:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829975.234466:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829975/real 1787829975] req@ffff9e680a2d61c0 x1874668790828288/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829975.234473:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790828288/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829975.234478:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829975.234480:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829975.234481:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790828288:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829975.234484:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790828416:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829975.234492:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829975.234497:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829975/real 1787829975] req@ffff9e680a2d52c0 x1874668790828416/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829975.234503:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790828416/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829975.234507:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829975.234509:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829975.234510:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790828416:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829975.234513:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790828544:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829975.234521:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829975.234525:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829975/real 1787829975] req@ffff9e680a2d7480 x1874668790828544/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829975.234531:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790828544/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829975.234539:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829975.234541:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829975.234543:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790828544:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829975.234545:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790828672:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829975.234553:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829975.234558:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829975/real 1787829975] req@ffff9e680a2d4b40 x1874668790828672/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829975.234564:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790828672/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829975.234568:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829975.234570:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829975.234571:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790828672:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829975.745831:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829975.938118:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829976.770338:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829976.962000:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829977.793818:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829977.985868:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829978.817789:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:0.0:1787829979.009941:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829979.841869:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829980.035217:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829980.354297:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829980.354308:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829980.354313:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829980.354323:0:1377:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6617 00000100:00080000:1.0:1787829980.354333:0:1377:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829980.354340:0:1377:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829980.354391:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790828800/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829980.354469:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790828800:192.168.203.109@tcp:250:kworker.0 00000800:00000100:3.0:1787829980.354517:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:1.0:1787829980.354599:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829980.354602:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829980.354604:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829980.354607:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6617 00000100:00080000:1.0:1787829980.354610:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829980.354613:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829980.354620:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829980.354630:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790828928/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829980.354686:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829980/real 1787829980] req@ffff9e6831aa8f00 x1874668790828800/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829980.354714:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790828800/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829980.354721:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829980.354723:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829980.354726:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790828800:192.168.203.109@tcp:250:kworker.0 00000100:00100000:3.0:1787829980.354744:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790828928:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829980.354760:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829980.354768:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829980/real 1787829980] req@ffff9e6831aa9e00 x1874668790828928/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829980.354774:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790828928/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829980.354780:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829980.354781:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829980.354783:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790828928:192.168.203.109@tcp:38:kworker.0 00000100:00080000:1.0:1787829980.354957:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829980.354961:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829980.354963:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829980.354968:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6617 00000100:00080000:1.0:1787829980.354972:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829980.354976:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829980.354981:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829980.354993:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8b40 x1874668790829056/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829980.355014:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829980.355018:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829980.355019:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829980.355023:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6617 00000100:00080000:1.0:1787829980.355026:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829980.355030:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829980.355036:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829980.355048:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aab480 x1874668790829184/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829980.355057:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829980.355060:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829980.355061:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829980.355064:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6617 00000100:00080000:1.0:1787829980.355068:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829980.355071:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829980.355076:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829980.355086:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa9e00 x1874668790829312/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829980.355094:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829980.355097:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829980.355099:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829980.355102:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6617 00000100:00080000:1.0:1787829980.355104:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829980.355108:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829980.355112:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829980.355120:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aa8f00 x1874668790829440/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829980.355132:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829980.355135:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787829980.355137:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829980.355139:0:1377:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6617 00000100:00080000:1.0:1787829980.355142:0:1377:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829980.355146:0:1377:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829980.355150:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829980.355158:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa580 x1874668790829568/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829980.355166:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829980.355169:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829980.355171:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829980.355174:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6617 00000100:00080000:1.0:1787829980.355177:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829980.355181:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829980.355185:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829980.355194:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaad00 x1874668790829696/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829980.355203:0:1377:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829980.355206:0:1377:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829980.355207:0:1377:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829980.355211:0:1377:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6617 00000100:00080000:1.0:1787829980.355214:0:1377:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829980.355217:0:1377:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:1.0:1787829980.355222:0:1377:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829980.355230:0:1377:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6831aaa1c0 x1874668790829824/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829980.355301:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790829056:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829980.355324:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829980.355336:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829980/real 1787829980] req@ffff9e6831aa8b40 x1874668790829056/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829980.355347:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8b40 x1874668790829056/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829980.355355:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829980.355358:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829980.355361:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790829056:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829980.355366:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790829184:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829980.355387:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829980.355398:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829980/real 1787829980] req@ffff9e6831aab480 x1874668790829184/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829980.355408:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aab480 x1874668790829184/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829980.355415:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829980.355418:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829980.355421:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aab480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790829184:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829980.355426:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790829312:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829980.355444:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829980.355453:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829980/real 1787829980] req@ffff9e6831aa9e00 x1874668790829312/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829980.355468:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa9e00 x1874668790829312/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829980.355474:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829980.355476:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829980.355479:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa9e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790829312:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829980.355483:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790829440:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829980.355499:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829980.355509:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829980/real 1787829980] req@ffff9e6831aa8f00 x1874668790829440/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829980.355519:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aa8f00 x1874668790829440/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829980.355528:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829980.355530:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829980.355533:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aa8f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790829440:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829980.355538:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790829568:192.168.203.109@tcp:38:kworker.0 00000800:00000100:3.0:1787829980.355556:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829980.355566:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829980/real 1787829980] req@ffff9e6831aaa580 x1874668790829568/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829980.355576:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa580 x1874668790829568/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829980.355589:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829980.355592:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829980.355595:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790829568:192.168.203.109@tcp:38:kworker.0 00000100:00100000:3.0:1787829980.355601:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790829696:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829980.355620:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829980.355630:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829980/real 1787829980] req@ffff9e6831aaad00 x1874668790829696/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829980.355639:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaad00 x1874668790829696/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829980.355646:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829980.355648:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829980.355651:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790829696:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787829980.355657:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790829824:192.168.203.109@tcp:8:kworker.0 00000800:00000100:3.0:1787829980.355673:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:3.0:1787829980.355682:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829980/real 1787829980] req@ffff9e6831aaa1c0 x1874668790829824/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829980.355691:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6831aaa1c0 x1874668790829824/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829980.355716:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829980.355719:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:3.0:1787829980.355722:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6831aaa1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790829824:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829980.865809:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829981.058926:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829981.889857:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829982.081953:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829982.913938:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829983.105904:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829983.938095:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829984.129880:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829984.961817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829985.154036:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829985.474202:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829985.474210:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829985.474214:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829985.474223:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6622 00000100:00080000:3.0:1787829985.474233:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829985.474244:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787829985.474285:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790829952/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829985.474310:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829985.474314:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829985.474316:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829985.474320:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6622 00000100:00080000:3.0:1787829985.474324:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829985.474328:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829985.474344:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829985.474358:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790830080/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829985.474366:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829985.474369:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829985.474371:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829985.474374:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6622 00000100:00080000:3.0:1787829985.474377:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829985.474381:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829985.474385:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829985.474393:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790830208/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829985.474402:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829985.474405:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829985.474407:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829985.474410:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6622 00000100:00080000:3.0:1787829985.474413:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829985.474417:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829985.474422:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829985.474441:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790830336/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829985.474449:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829985.474451:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829985.474453:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829985.474456:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6622 00000100:00080000:3.0:1787829985.474459:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829985.474461:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829985.474465:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829985.474472:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790830464/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829985.474485:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829985.474555:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829985.474557:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829985.474561:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6622 00000100:00080000:3.0:1787829985.474565:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829985.474569:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829985.474574:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829985.474586:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790830592/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829985.474596:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829985.474599:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787829985.474600:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829985.474616:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6622 00000100:00080000:3.0:1787829985.474620:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829985.474623:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829985.474633:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829985.474643:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790830720/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829985.474651:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829985.474654:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829985.474656:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829985.474778:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6622 00000100:00080000:3.0:1787829985.474784:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829985.474789:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829985.474795:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829985.474819:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790830848/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829985.474828:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829985.474831:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829985.474833:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829985.474837:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6622 00000100:00080000:3.0:1787829985.474840:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829985.474844:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787829985.474850:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787829985.474861:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790830976/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829985.474971:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790829952:192.168.203.109@tcp:250:kworker.0 00000800:00000100:1.0:1787829985.475026:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829985.475061:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829985/real 1787829985] req@ffff9e680a2d4780 x1874668790829952/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829985.475076:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790829952/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829985.475086:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829985.475089:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829985.475093:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790829952:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787829985.475100:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790830080:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829985.475119:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829985.475127:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829985/real 1787829985] req@ffff9e680a2d70c0 x1874668790830080/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829985.475134:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790830080/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829985.475140:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829985.475142:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829985.475144:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790830080:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829985.475147:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790830208:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829985.475160:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829985.475170:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829985/real 1787829985] req@ffff9e680a2d6580 x1874668790830208/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829985.475181:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790830208/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829985.475188:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829985.475190:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829985.475193:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790830208:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829985.475199:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790830336:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829985.475217:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829985.475227:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829985/real 1787829985] req@ffff9e680a2d4000 x1874668790830336/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829985.475239:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790830336/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829985.475246:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829985.475249:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829985.475253:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790830336:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829985.475258:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790830464:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829985.475276:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829985.475287:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829985/real 1787829985] req@ffff9e680a2d43c0 x1874668790830464/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829985.475297:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790830464/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829985.475303:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829985.475306:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829985.475309:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790830464:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829985.475314:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790830592:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829985.475331:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829985.475342:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829985/real 1787829985] req@ffff9e680a2d4b40 x1874668790830592/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829985.475352:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790830592/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829985.475361:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829985.475364:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829985.475367:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790830592:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829985.475372:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790830720:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829985.475390:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829985.475401:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829985/real 1787829985] req@ffff9e680a2d7480 x1874668790830720/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829985.475412:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790830720/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829985.475420:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829985.475422:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829985.475425:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790830720:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829985.475431:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790830848:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829985.475450:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829985.475459:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829985/real 1787829985] req@ffff9e680a2d52c0 x1874668790830848/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829985.475469:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790830848/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829985.475476:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829985.475479:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829985.475482:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790830848:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829985.475488:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790830976:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829985.475505:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829985.475514:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829985/real 1787829985] req@ffff9e680a2d61c0 x1874668790830976/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829985.475525:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790830976/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829985.475532:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829985.475535:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829985.475537:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790830976:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829985.985841:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829986.177927:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829987.009810:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:0.0:1787829987.202081:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829988.033856:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:0.0:1787829988.228500:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829989.057810:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:0.0:1787829989.249985:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829990.081817:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829990.274006:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829990.593943:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829990.593953:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829990.593957:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829990.593965:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6627 00000100:00080000:0.0:1787829990.593975:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829990.593981:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829990.594013:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790831104/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829990.594037:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829990.594040:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829990.594042:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829990.594046:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6627 00000100:00080000:0.0:1787829990.594049:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829990.594052:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829990.594061:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829990.594072:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790831232/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829990.594083:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829990.594085:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790831104:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829990.594087:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829990.594090:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829990.594097:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6627 00000100:00080000:0.0:1787829990.594100:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829990.594103:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829990.594107:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829990.594118:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790831360/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829990.594126:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829990.594130:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829990.594132:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829990.594135:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6627 00000100:00080000:0.0:1787829990.594141:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829990.594144:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829990.594149:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787829990.594152:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787829990.594162:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790831488/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829990.594173:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829990.594176:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829990.594178:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829990.594181:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6627 00000100:00080000:0.0:1787829990.594184:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829990.594188:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829990.594192:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829990.594196:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829990/real 1787829990] req@ffff9e6803e9b840 x1874668790831104/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829990.594205:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790831616/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829990.594216:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829990.594219:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829990.594223:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829990.594223:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790831104/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829990.594230:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6627 00000100:00080000:0.0:1787829990.594236:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829990.594242:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787829990.594243:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787829990.594246:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787829990.594250:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829990.594256:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790831104:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787829990.594261:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790831744/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829990.594269:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829990.594272:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829990.594273:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829990.594278:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6627 00000100:00080000:0.0:1787829990.594280:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829990.594282:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790831232:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829990.594287:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829990.594293:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829990.594306:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790831872/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829990.594314:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829990.594320:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829990.594324:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829990.594326:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829990.594332:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6627 00000100:00100000:1.0:1787829990.594334:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829990/real 1787829990] req@ffff9e6803e9b480 x1874668790831232/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829990.594337:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829990.594345:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829990.594353:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829990.594358:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790831232/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829990.594366:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790832000/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829990.594371:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829990.594379:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829990.594380:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829990.594386:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829990.594386:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790831232:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829990.594390:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829990.594397:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6627 00000100:00100000:1.0:1787829990.594399:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790831360:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829990.594402:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829990.594408:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829990.594413:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829990.594426:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790832128/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829990.594427:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829990.594445:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829990/real 1787829990] req@ffff9e6803e99e00 x1874668790831360/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829990.594457:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790831360/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829990.594465:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829990.594469:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829990.594471:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790831360:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829990.594477:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790831488:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829990.594494:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829990.594503:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829990/real 1787829990] req@ffff9e6803e99680 x1874668790831488/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829990.594513:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790831488/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829990.594520:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829990.594523:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829990.594525:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790831488:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829990.594531:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790831616:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829990.594547:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829990.594556:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829990/real 1787829990] req@ffff9e6803e9a580 x1874668790831616/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829990.594566:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790831616/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829990.594573:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829990.594576:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829990.594579:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790831616:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829990.594593:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790831744:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829990.594615:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829990.594628:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829990/real 1787829990] req@ffff9e6803e9a1c0 x1874668790831744/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829990.594640:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790831744/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829990.594648:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829990.594651:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829990.594655:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790831744:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829990.594674:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790831872:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829990.594693:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829990.594725:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829990/real 1787829990] req@ffff9e6803e98780 x1874668790831872/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829990.594742:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790831872/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829990.594749:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829990.594751:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829990.594754:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790831872:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829990.594759:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790832000:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829990.594797:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829990.594807:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829990/real 1787829990] req@ffff9e6803e98b40 x1874668790832000/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829990.594818:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790832000/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829990.594825:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829990.594828:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829990.594830:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790832000:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829990.594836:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790832128:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829990.594852:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829990.594860:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829990/real 1787829990] req@ffff9e6803e992c0 x1874668790832128/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829990.594870:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790832128/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829990.594881:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829990.594884:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829990.594887:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790832128:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829991.105849:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:0.0:1787829991.298232:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829992.129801:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:0.0:1787829992.323721:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829993.153836:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829993.346317:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829994.177803:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829994.369951:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829995.201866:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829995.393848:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829995.713799:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829995.713808:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829995.713812:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829995.713821:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6632 00000100:00080000:0.0:1787829995.713831:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829995.713837:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787829995.713868:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790832256/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829995.713893:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829995.713896:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829995.713898:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829995.713901:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6632 00000100:00080000:0.0:1787829995.713904:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829995.713907:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829995.713915:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829995.713926:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790832384/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829995.713936:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790832256:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829995.713939:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829995.713945:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829995.713947:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829995.713951:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6632 00000100:00080000:0.0:1787829995.713954:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829995.713958:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829995.713962:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829995.713973:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790832512/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829995.713982:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829995.713985:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829995.713987:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829995.713991:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6632 00000100:00080000:0.0:1787829995.713994:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829995.713997:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1787829995.714002:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787829995.714004:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829995.714021:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790832640/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829995.714031:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829995.714034:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829995.714036:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829995.714041:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6632 00000100:00080000:0.0:1787829995.714045:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829995.714046:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829995/real 1787829995] req@ffff9e6803e98f00 x1874668790832256/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829995.714049:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829995.714056:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829995.714068:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790832256/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829995.714069:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790832768/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829995.714096:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829995.714103:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829995.714105:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829995.714108:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829995.714112:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829995.714112:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790832256:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787829995.714118:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6632 00000100:00080000:0.0:1787829995.714122:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829995.714126:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787829995.714130:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829995.714141:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790832384:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787829995.714142:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790832896/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829995.714156:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829995.714159:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787829995.714162:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829995.714165:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6632 00000100:00080000:0.0:1787829995.714169:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829995.714175:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1787829995.714175:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787829995.714182:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829995.714194:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829995/real 1787829995] req@ffff9e6803e9a940 x1874668790832384/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829995.714195:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790833024/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829995.714222:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829995.714226:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790832384/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829995.714228:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829995.714231:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829995.714239:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829995.714240:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6632 00000100:00080000:1.0:1787829995.714245:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787829995.714247:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829995.714248:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790832384:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787829995.714254:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787829995.714256:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790832512:192.168.203.109@tcp:38:kworker.0 00000008:00100000:0.0:1787829995.714261:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787829995.714274:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790833152/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829995.714278:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787829995.714288:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829995.714289:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829995/real 1787829995] req@ffff9e6803e9bc00 x1874668790832512/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829995.714296:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829995.714300:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829995.714304:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790832512/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829995.714307:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6632 00000100:00080000:0.0:1787829995.714313:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829995.714315:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829995.714319:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787829995.714322:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000008:00100000:0.0:1787829995.714326:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787829995.714327:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790832512:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829995.714337:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790832640:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787829995.714342:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790833280/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787829995.714362:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829995.714375:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829995/real 1787829995] req@ffff9e6803e9ad00 x1874668790832640/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829995.714384:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790832640/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829995.714389:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829995.714391:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829995.714393:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790832640:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829995.714397:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790832768:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829995.714410:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829995.714416:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829995/real 1787829995] req@ffff9e6803e992c0 x1874668790832768/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829995.714422:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790832768/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829995.714426:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829995.714428:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829995.714429:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790832768:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829995.714440:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790832896:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829995.714456:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829995.714462:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829995/real 1787829995] req@ffff9e6803e98b40 x1874668790832896/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829995.714469:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790832896/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829995.714475:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829995.714477:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829995.714478:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790832896:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829995.714481:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790833024:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787829995.714491:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829995.714495:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829995/real 1787829995] req@ffff9e6803e98780 x1874668790833024/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829995.714506:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790833024/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829995.714510:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829995.714511:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829995.714513:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790833024:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787829995.714516:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790833152:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829995.714524:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829995.714529:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829995/real 1787829995] req@ffff9e6803e9a1c0 x1874668790833152/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829995.714534:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790833152/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829995.714539:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829995.714540:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829995.714542:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790833152:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787829995.714545:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790833280:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787829995.714553:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787829995.714558:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829995/real 1787829995] req@ffff9e6803e9a580 x1874668790833280/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829995.714564:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790833280/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829995.714571:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829995.714573:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787829995.714575:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790833280:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787829996.225777:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829996.417876:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829997.249859:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829997.441870:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829998.273848:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787829998.465900:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829999.297797:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787829999.489902:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830000.321848:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787830000.513922:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830000.833817:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830000.833829:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830000.833834:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830000.833846:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6637 00000100:00080000:0.0:1787830000.833858:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830000.833866:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787830000.833913:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99680 x1874668790833408/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830000.833941:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830000.833947:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830000.833949:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830000.833952:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6637 00000100:00080000:0.0:1787830000.833955:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830000.833958:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787830000.833969:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830000.833978:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e99e00 x1874668790833536/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830000.833988:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830000.833991:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830000.833994:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830000.833997:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6637 00000100:00080000:0.0:1787830000.833999:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830000.834003:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787830000.834007:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830000.834014:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b480 x1874668790833664/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830000.834022:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830000.834025:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830000.834028:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830000.834035:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6637 00000100:00080000:0.0:1787830000.834040:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787830000.834040:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790833408:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787830000.834046:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787830000.834051:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830000.834059:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9b840 x1874668790833792/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830000.834079:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830000.834082:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830000.834083:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830000.834087:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6637 00000100:00080000:0.0:1787830000.834090:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830000.834094:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787830000.834100:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830000.834114:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a580 x1874668790833920/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830000.834122:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830000.834125:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830000.834127:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830000.834133:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6637 00000100:00080000:0.0:1787830000.834137:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830000.834143:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000800:00000100:1.0:1787830000.834147:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000008:00100000:0.0:1787830000.834151:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830000.834167:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a1c0 x1874668790834048/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830000.834176:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830000.834180:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830000.834182:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830000.834187:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6637 00000100:00080000:0.0:1787830000.834191:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830000.834196:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787830000.834203:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830000/real 1787830000] req@ffff9e6803e99680 x1874668790833408/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787830000.834206:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830000.834227:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790834176/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830000.834237:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830000.834245:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787830000.834245:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99680 x1874668790833408/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830000.834259:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830000.834262:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830000.834271:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6637 00000100:00080000:1.0:1787830000.834276:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:0.0:1787830000.834278:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787830000.834284:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790833408:192.168.203.109@tcp:250:kworker.0 00000100:00080000:0.0:1787830000.834289:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787830000.834297:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787830000.834299:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790833536:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787830000.834317:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790834304/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830000.834328:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830000.834332:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787830000.834336:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787830000.834338:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830000.834346:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6637 00000100:00080000:0.0:1787830000.834351:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787830000.834358:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830000/real 1787830000] req@ffff9e6803e99e00 x1874668790833536/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830000.834359:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787830000.834367:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830000.834397:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790834432/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830000.834398:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e99e00 x1874668790833536/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830000.834434:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830000.834440:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830000.834444:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e99e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790833536:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787830000.834474:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790833664:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787830000.834496:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787830000.834507:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830000/real 1787830000] req@ffff9e6803e9b480 x1874668790833664/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830000.834517:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b480 x1874668790833664/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830000.834525:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830000.834528:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830000.834531:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790833664:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787830000.834538:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790833792:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787830000.834557:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787830000.834568:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830000/real 1787830000] req@ffff9e6803e9b840 x1874668790833792/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830000.834580:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9b840 x1874668790833792/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830000.834588:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830000.834592:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830000.834595:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790833792:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787830000.834602:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790833920:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787830000.834621:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787830000.834632:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830000/real 1787830000] req@ffff9e6803e9a580 x1874668790833920/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830000.834643:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a580 x1874668790833920/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830000.834651:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830000.834654:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830000.834657:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790833920:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787830000.834662:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790834048:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787830000.834680:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787830000.834691:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830000/real 1787830000] req@ffff9e6803e9a1c0 x1874668790834048/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830000.834747:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a1c0 x1874668790834048/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830000.834756:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830000.834759:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830000.834762:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790834048:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787830000.834768:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790834176:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787830000.834789:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787830000.834803:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830000/real 1787830000] req@ffff9e6803e98780 x1874668790834176/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830000.834821:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790834176/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830000.834829:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830000.834832:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830000.834835:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790834176:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787830000.834840:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790834304:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787830000.834860:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787830000.834869:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830000/real 1787830000] req@ffff9e6803e98b40 x1874668790834304/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830000.834880:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790834304/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830000.834888:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830000.834891:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830000.834894:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790834304:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787830000.834900:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790834432:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787830000.834917:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787830000.834927:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830000/real 1787830000] req@ffff9e6803e992c0 x1874668790834432/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830000.834937:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790834432/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830000.834951:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830000.834954:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830000.834956:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790834432:192.168.203.109@tcp:8:kworker.0 00000100:00100000:2.0:1787830001.346580:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787830001.538018:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830002.369926:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787830002.561909:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830003.394070:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787830003.586218:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830004.417855:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787830004.609988:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830005.442080:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787830005.633921:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830005.954068:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830005.954078:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830005.954083:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830005.954093:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6642 00000100:00080000:3.0:1787830005.954103:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830005.954111:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787830005.954147:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790834560/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830005.954183:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830005.954186:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830005.954188:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830005.954192:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6642 00000100:00080000:3.0:1787830005.954195:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830005.954198:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787830005.954207:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830005.954218:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790834688/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830005.954227:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830005.954230:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830005.954231:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830005.954235:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6642 00000100:00080000:3.0:1787830005.954237:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830005.954241:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787830005.954244:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830005.954252:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790834816/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830005.954259:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830005.954262:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830005.954265:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830005.954268:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6642 00000100:00080000:3.0:1787830005.954270:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830005.954273:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787830005.954278:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830005.954285:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790834944/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830005.954293:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830005.954295:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830005.954297:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830005.954300:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6642 00000100:00080000:3.0:1787830005.954303:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830005.954306:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787830005.954309:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830005.954316:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790835072/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830005.954323:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830005.954326:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830005.954327:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830005.954330:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6642 00000100:00080000:3.0:1787830005.954333:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830005.954336:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787830005.954340:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830005.954351:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790835200/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830005.954359:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830005.954362:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830005.954363:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830005.954366:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6642 00000100:00080000:3.0:1787830005.954369:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830005.954372:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787830005.954375:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830005.954383:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790835328/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830005.954392:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830005.954394:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830005.954396:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830005.954398:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6642 00000100:00080000:3.0:1787830005.954401:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830005.954404:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787830005.954408:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830005.954415:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790835456/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830005.954423:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830005.954426:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830005.954427:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830005.954430:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6642 00000100:00080000:3.0:1787830005.954433:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830005.954436:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787830005.954440:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830005.954447:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790835584/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830005.954540:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790834560:192.168.203.109@tcp:250:kworker.0 00000800:00000100:1.0:1787830005.954596:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787830005.954631:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830005/real 1787830005] req@ffff9e680a2d4f00 x1874668790834560/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830005.954646:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790834560/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830005.954655:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830005.954659:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830005.954674:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790834560:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787830005.954681:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790834688:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787830005.954721:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787830005.954732:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830005/real 1787830005] req@ffff9e680a2d6d00 x1874668790834688/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830005.954742:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790834688/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830005.954756:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830005.954758:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830005.954762:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790834688:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787830005.954767:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790834816:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787830005.954782:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787830005.954790:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830005/real 1787830005] req@ffff9e680a2d5a40 x1874668790834816/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830005.954800:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790834816/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830005.954807:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830005.954810:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830005.954813:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790834816:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787830005.954818:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790834944:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787830005.954832:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787830005.954840:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830005/real 1787830005] req@ffff9e680a2d5e00 x1874668790834944/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830005.954851:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790834944/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830005.954857:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830005.954860:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830005.954863:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790834944:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787830005.954868:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790835072:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787830005.954881:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787830005.954889:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830005/real 1787830005] req@ffff9e680a2d7c00 x1874668790835072/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830005.954900:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790835072/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830005.954908:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830005.954911:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830005.954914:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790835072:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787830005.954918:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790835200:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787830005.954932:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787830005.954940:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830005/real 1787830005] req@ffff9e680a2d61c0 x1874668790835200/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830005.954950:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790835200/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830005.954959:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830005.954962:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830005.954965:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790835200:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787830005.954970:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790835328:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787830005.954983:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787830005.954991:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830005/real 1787830005] req@ffff9e680a2d52c0 x1874668790835328/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830005.955002:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790835328/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830005.955009:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830005.955012:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830005.955015:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790835328:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787830005.955020:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790835456:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787830005.955034:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787830005.955042:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830005/real 1787830005] req@ffff9e680a2d7480 x1874668790835456/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830005.955052:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790835456/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830005.955060:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830005.955062:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830005.955065:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790835456:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787830005.955070:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790835584:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787830005.955083:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787830005.955091:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830005/real 1787830005] req@ffff9e680a2d4b40 x1874668790835584/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830005.955102:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790835584/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830005.955108:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830005.955111:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830005.955114:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790835584:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787830006.465831:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787830006.658570:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830007.492384:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787830007.682608:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830008.514208:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787830008.706637:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830009.538121:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787830009.729931:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830010.561870:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787830010.754374:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830011.073879:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830011.073889:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830011.073892:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830011.073902:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6647 00000100:00080000:0.0:1787830011.073912:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830011.073919:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:0.0:1787830011.073952:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790835712/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830011.074029:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790835712:192.168.203.109@tcp:250:kworker.0 00000800:00000100:1.0:1787830011.074086:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:0.0:1787830011.074115:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830011.074119:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830011.074121:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830011.074126:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6647 00000100:00080000:0.0:1787830011.074130:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830011.074135:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787830011.074144:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830011.074158:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790835840/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830011.074218:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830011/real 1787830011] req@ffff9e6803e9ad00 x1874668790835712/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830011.074234:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790835712/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830011.074244:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830011.074247:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830011.074251:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790835712:192.168.203.109@tcp:250:kworker.0 00000100:00100000:1.0:1787830011.074274:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790835840:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787830011.074298:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787830011.074311:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830011/real 1787830011] req@ffff9e6803e9bc00 x1874668790835840/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830011.074322:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790835840/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830011.074337:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830011.074340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830011.074343:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790835840:192.168.203.109@tcp:38:kworker.0 00000100:00080000:0.0:1787830011.074394:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830011.074398:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830011.074400:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830011.074404:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6647 00000100:00080000:0.0:1787830011.074408:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830011.074413:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787830011.074418:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830011.074431:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9a940 x1874668790835968/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830011.074440:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830011.074443:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830011.074444:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830011.074448:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6647 00000100:00080000:0.0:1787830011.074452:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830011.074455:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787830011.074461:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830011.074470:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98f00 x1874668790836096/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830011.074478:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830011.074481:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830011.074482:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830011.074486:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6647 00000100:00080000:0.0:1787830011.074489:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830011.074492:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787830011.074496:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830011.074506:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9bc00 x1874668790836224/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830011.074514:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830011.074517:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830011.074519:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830011.074523:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6647 00000100:00080000:0.0:1787830011.074526:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830011.074530:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787830011.074534:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830011.074543:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e9ad00 x1874668790836352/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830011.074551:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830011.074554:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787830011.074555:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830011.074559:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6647 00000100:00080000:0.0:1787830011.074562:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830011.074565:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787830011.074569:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830011.074579:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e992c0 x1874668790836480/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830011.074587:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830011.074590:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830011.074591:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830011.074594:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6647 00000100:00080000:0.0:1787830011.074597:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830011.074601:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787830011.074606:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830011.074614:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98b40 x1874668790836608/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830011.074623:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787830011.076692:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790835968:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787830011.076750:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787830011.076768:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830011/real 1787830011] req@ffff9e6803e9a940 x1874668790835968/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830011.076782:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9a940 x1874668790835968/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830011.076790:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830011.076794:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830011.076797:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9a940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790835968:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787830011.076804:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790836096:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787830011.076825:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787830011.076836:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830011/real 1787830011] req@ffff9e6803e98f00 x1874668790836096/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830011.076848:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98f00 x1874668790836096/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830011.076855:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830011.076858:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830011.076862:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790836096:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787830011.076868:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790836224:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787830011.076885:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787830011.076896:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830011/real 1787830011] req@ffff9e6803e9bc00 x1874668790836224/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830011.076906:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9bc00 x1874668790836224/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830011.076914:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830011.076918:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830011.076921:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790836224:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787830011.076926:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790836352:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787830011.076945:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787830011.076955:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830011/real 1787830011] req@ffff9e6803e9ad00 x1874668790836352/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830011.076966:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e9ad00 x1874668790836352/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830011.076976:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830011.076979:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830011.076982:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e9ad00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790836352:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787830011.076987:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790836480:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787830011.077004:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787830011.077015:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830011/real 1787830011] req@ffff9e6803e992c0 x1874668790836480/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830011.077026:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e992c0 x1874668790836480/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830011.077033:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830011.077036:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830011.077039:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e992c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790836480:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787830011.077044:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790836608:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787830011.077061:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787830011.077070:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830011/real 1787830011] req@ffff9e6803e98b40 x1874668790836608/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830011.077081:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98b40 x1874668790836608/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830011.077089:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830011.077092:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830011.077095:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790836608:192.168.203.109@tcp:8:kworker.0 00000100:00080000:0.0:1787830011.077175:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830011.077178:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830011.077183:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6647 00000100:00080000:0.0:1787830011.077187:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830011.077192:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:0.0:1787830011.077199:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830011.077214:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6803e98780 x1874668790836736/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830011.077290:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790836736:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787830011.077313:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787830011.077325:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830011/real 1787830011] req@ffff9e6803e98780 x1874668790836736/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830011.077336:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6803e98780 x1874668790836736/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830011.077344:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830011.077347:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830011.077350:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6803e98780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790836736:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787830011.585804:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787830011.779716:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830012.609848:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787830012.802891:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830013.635143:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787830013.826163:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830014.657865:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787830014.850247:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830015.681832:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787830015.874288:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830016.193828:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830016.193837:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830016.193842:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830016.193851:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6652 00000100:00080000:2.0:1787830016.193862:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830016.193869:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787830016.193903:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790836864/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830016.193935:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830016.193939:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830016.193941:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830016.193944:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6652 00000100:00080000:2.0:1787830016.193948:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830016.193952:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787830016.193960:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830016.193971:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790836992/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830016.193980:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830016.193983:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830016.193984:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830016.193987:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6652 00000100:00080000:2.0:1787830016.193991:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830016.193994:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787830016.193998:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830016.194020:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790837120/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830016.194031:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790836864:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787830016.194041:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830016.194045:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830016.194047:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830016.194051:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6652 00000100:00080000:2.0:1787830016.194055:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830016.194058:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787830016.194063:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830016.194075:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790837248/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830016.194084:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830016.194088:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830016.194089:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830016.194094:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6652 00000800:00000100:1.0:1787830016.194096:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787830016.194101:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830016.194107:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787830016.194112:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830016.194124:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790837376/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830016.194133:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787830016.194139:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830016/real 1787830016] req@ffff9e6804421680 x1874668790836864/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830016.194139:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830016.194143:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830016.194151:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6652 00000100:00080000:2.0:1787830016.194158:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830016.194163:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:1.0:1787830016.194168:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790836864/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787830016.194169:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787830016.194183:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787830016.194186:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790837504/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830016.194189:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830016.194197:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790836864:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787830016.194202:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830016.194207:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1787830016.194208:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790836992:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787830016.194210:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830016.194218:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6652 00000100:00080000:2.0:1787830016.194222:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830016.194226:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787830016.194230:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787830016.194239:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787830016.194244:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790837632/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830016.194255:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830016/real 1787830016] req@ffff9e6804420b40 x1874668790836992/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830016.194255:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830016.194263:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830016.194267:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830016.194275:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6652 00000100:00100000:1.0:1787830016.194279:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790836992/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830016.194282:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830016.194291:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:1.0:1787830016.194293:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787830016.194297:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787830016.194300:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830016.194305:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790836992:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787830016.194312:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790837760/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830016.194327:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830016.194330:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830016.194331:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830016.194334:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6652 00000100:00080000:2.0:1787830016.194337:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830016.194340:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787830016.194344:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830016.194351:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790837888/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830016.194394:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790837120:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787830016.194415:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787830016.194426:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830016/real 1787830016] req@ffff9e68044230c0 x1874668790837120/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830016.194438:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790837120/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830016.194445:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830016.194448:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830016.194451:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790837120:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787830016.194456:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790837248:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787830016.194472:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787830016.194481:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830016/real 1787830016] req@ffff9e68044212c0 x1874668790837248/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830016.194491:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790837248/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830016.194498:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830016.194501:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830016.194504:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790837248:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787830016.194509:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790837376:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787830016.194523:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787830016.194533:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830016/real 1787830016] req@ffff9e6804421e00 x1874668790837376/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830016.194543:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790837376/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830016.194550:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830016.194553:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830016.194556:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790837376:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787830016.194561:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790837504:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787830016.194577:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787830016.194585:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830016/real 1787830016] req@ffff9e6804422580 x1874668790837504/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830016.194613:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790837504/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830016.194621:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830016.194624:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830016.194627:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790837504:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787830016.194632:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790837632:192.168.203.109@tcp:38:kworker.0 00000800:00000100:1.0:1787830016.194650:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787830016.194661:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830016/real 1787830016] req@ffff9e6804420f00 x1874668790837632/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830016.194677:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790837632/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830016.194683:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830016.194685:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830016.194688:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790837632:192.168.203.109@tcp:38:kworker.0 00000100:00100000:1.0:1787830016.194693:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790837760:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787830016.194737:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787830016.194748:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830016/real 1787830016] req@ffff9e6804423480 x1874668790837760/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830016.194758:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790837760/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830016.194764:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830016.194767:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830016.194770:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790837760:192.168.203.109@tcp:8:kworker.0 00000100:00100000:1.0:1787830016.194775:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790837888:192.168.203.109@tcp:8:kworker.0 00000800:00000100:1.0:1787830016.194792:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:1.0:1787830016.194802:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830016/real 1787830016] req@ffff9e6804423840 x1874668790837888/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830016.194812:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790837888/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830016.194824:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830016.194826:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:1.0:1787830016.194829:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790837888:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787830016.705835:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787830016.898955:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830017.730727:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:0.0:1787830017.921978:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830018.753826:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787830018.947984:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830019.778033:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787830019.970032:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830020.801839:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787830020.993954:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830021.315807:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830021.315816:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830021.315820:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830021.315829:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6658 00000100:00080000:3.0:1787830021.315838:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830021.315845:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787830021.315874:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790838016/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830021.315908:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830021.315912:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830021.315913:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830021.315917:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6658 00000100:00080000:3.0:1787830021.315920:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830021.315924:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787830021.315934:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830021.315944:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790838144/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830021.315953:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830021.315956:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830021.315957:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830021.315961:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6658 00000100:00080000:3.0:1787830021.315963:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830021.315967:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787830021.315971:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830021.315978:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790838272/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830021.315987:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830021.315990:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830021.315991:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830021.315994:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6658 00000100:00080000:3.0:1787830021.315997:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830021.316000:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787830021.316005:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830021.316013:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790838400/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830021.316021:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830021.316024:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830021.316026:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830021.316029:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6658 00000100:00080000:3.0:1787830021.316032:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830021.316036:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787830021.316040:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830021.316048:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790838528/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830021.316056:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830021.316058:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830021.316060:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830021.316063:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6658 00000100:00080000:3.0:1787830021.316066:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830021.316069:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787830021.316073:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830021.316081:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790838656/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830021.316089:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830021.316092:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830021.316093:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830021.316096:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6658 00000100:00080000:3.0:1787830021.316099:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830021.316102:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787830021.316106:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830021.316114:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790838784/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830021.316120:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830021.316123:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830021.316125:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830021.316128:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6658 00000100:00080000:3.0:1787830021.316130:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830021.316133:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787830021.316136:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830021.316143:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790838912/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830021.316157:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830021.316160:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830021.316161:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830021.316164:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6658 00000100:00080000:3.0:1787830021.316167:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830021.316170:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787830021.316174:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830021.316181:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790839040/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830021.316476:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790838016:192.168.203.109@tcp:250:kworker.0 00000800:00000100:0.0:1787830021.316525:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787830021.316569:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830021/real 1787830021] req@ffff9e680a2d43c0 x1874668790838016/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830021.316584:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790838016/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830021.316594:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830021.316598:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830021.316601:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790838016:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787830021.316608:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790838144:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787830021.316627:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787830021.316638:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830021/real 1787830021] req@ffff9e680a2d4000 x1874668790838144/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830021.316649:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790838144/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830021.316656:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830021.316659:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830021.316662:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790838144:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787830021.316668:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790838272:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787830021.316682:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787830021.316690:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830021/real 1787830021] req@ffff9e680a2d6580 x1874668790838272/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830021.316720:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790838272/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830021.316728:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830021.316731:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830021.316734:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790838272:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787830021.316739:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790838400:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787830021.316755:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787830021.316763:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830021/real 1787830021] req@ffff9e680a2d70c0 x1874668790838400/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830021.316780:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790838400/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830021.316787:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830021.316789:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830021.316792:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790838400:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787830021.316797:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790838528:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787830021.316811:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787830021.316819:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830021/real 1787830021] req@ffff9e680a2d4780 x1874668790838528/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830021.316829:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790838528/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830021.316837:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830021.316840:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830021.316842:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790838528:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787830021.316847:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790838656:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787830021.316860:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787830021.316868:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830021/real 1787830021] req@ffff9e680a2d4b40 x1874668790838656/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830021.316878:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790838656/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830021.316892:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830021.316895:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830021.316898:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790838656:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787830021.316902:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790838784:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787830021.316916:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787830021.316924:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830021/real 1787830021] req@ffff9e680a2d7480 x1874668790838784/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830021.316935:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790838784/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830021.316942:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830021.316944:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830021.316947:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790838784:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787830021.316953:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790838912:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787830021.316966:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787830021.316973:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830021/real 1787830021] req@ffff9e680a2d52c0 x1874668790838912/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830021.316984:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790838912/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830021.316991:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830021.316994:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830021.316998:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790838912:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787830021.317003:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790839040:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787830021.317018:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787830021.317027:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830021/real 1787830021] req@ffff9e680a2d61c0 x1874668790839040/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830021.317038:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790839040/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830021.317045:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830021.317048:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830021.317051:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790839040:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787830021.825904:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:0.0:1787830022.018322:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830022.849838:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787830023.044831:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830023.873796:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787830024.069572:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830024.897835:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787830025.090565:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830025.921854:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:0.0:1787830026.114564:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830026.433892:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830026.433906:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830026.433911:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830026.433924:0:89028:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6663 00000100:00080000:3.0:1787830026.433938:0:89028:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830026.433948:0:89028:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787830026.433997:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790839168/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830026.434082:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790839168:192.168.203.109@tcp:250:kworker.0 00000800:00000100:0.0:1787830026.434162:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787830026.434199:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830026.434204:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830026.434205:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830026.434210:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6663 00000100:00080000:3.0:1787830026.434214:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830026.434219:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787830026.434230:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830026.434245:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790839296/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830026.434312:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830026/real 1787830026] req@ffff9e680a2d7c00 x1874668790839168/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830026.434331:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790839168/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830026.434343:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830026.434348:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830026.434352:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790839168:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787830026.434384:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790839296:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787830026.434409:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787830026.434422:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830026/real 1787830026] req@ffff9e680a2d5e00 x1874668790839296/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830026.434434:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790839296/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830026.434443:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830026.434447:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830026.434450:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790839296:192.168.203.109@tcp:38:kworker.0 00000100:00080000:3.0:1787830026.434558:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830026.434563:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830026.434565:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830026.434569:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6663 00000100:00080000:3.0:1787830026.434574:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830026.434578:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787830026.434583:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830026.434597:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790839424/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830026.434639:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790839424:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787830026.434661:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:3.0:1787830026.434682:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830026.434686:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830026.434688:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830026.434714:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6663 00000100:00080000:3.0:1787830026.434720:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830026.434724:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787830026.434730:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830026.434742:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790839552/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830026.434803:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830026/real 1787830026] req@ffff9e680a2d5a40 x1874668790839424/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830026.434816:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790839424/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830026.434825:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830026.434828:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830026.434831:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790839424:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787830026.434845:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790839552:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787830026.434866:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787830026.434877:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830026/real 1787830026] req@ffff9e680a2d6d00 x1874668790839552/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830026.434895:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790839552/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830026.434903:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830026.434906:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830026.434909:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790839552:192.168.203.109@tcp:8:kworker.0 00000100:00080000:3.0:1787830026.434990:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830026.434994:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830026.434996:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830026.435000:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6663 00000100:00080000:3.0:1787830026.435004:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830026.435009:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787830026.435014:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830026.435027:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4f00 x1874668790839680/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830026.435037:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830026.435041:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830026.435043:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830026.435047:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6663 00000100:00080000:3.0:1787830026.435050:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830026.435054:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787830026.435058:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830026.435069:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6d00 x1874668790839808/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830026.435078:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830026.435081:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830026.435083:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830026.435087:0:89028:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6663 00000100:00080000:3.0:1787830026.435090:0:89028:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830026.435093:0:89028:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787830026.435098:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830026.435105:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5a40 x1874668790839936/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830026.435113:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830026.435116:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830026.435118:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830026.435122:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6663 00000100:00080000:3.0:1787830026.435125:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830026.435130:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787830026.435134:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830026.435143:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d5e00 x1874668790840064/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830026.435151:0:89028:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830026.435154:0:89028:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830026.435156:0:89028:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830026.435160:0:89028:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6663 00000100:00080000:3.0:1787830026.435164:0:89028:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830026.435167:0:89028:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787830026.435171:0:89028:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830026.435179:0:89028:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7c00 x1874668790840192/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830026.435243:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790839680:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787830026.435266:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787830026.435277:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830026/real 1787830026] req@ffff9e680a2d4f00 x1874668790839680/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830026.435288:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4f00 x1874668790839680/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830026.435297:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830026.435300:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830026.435303:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790839680:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787830026.435308:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790839808:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787830026.435327:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787830026.435337:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830026/real 1787830026] req@ffff9e680a2d6d00 x1874668790839808/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830026.435347:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6d00 x1874668790839808/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830026.435362:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830026.435365:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830026.435368:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790839808:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787830026.435374:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790839936:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787830026.435390:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787830026.435399:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830026/real 1787830026] req@ffff9e680a2d5a40 x1874668790839936/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830026.435411:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5a40 x1874668790839936/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830026.435418:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830026.435420:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830026.435423:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790839936:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787830026.435427:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790840064:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787830026.435441:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787830026.435448:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830026/real 1787830026] req@ffff9e680a2d5e00 x1874668790840064/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830026.435456:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d5e00 x1874668790840064/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830026.435464:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830026.435467:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830026.435470:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790840064:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787830026.435474:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790840192:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787830026.435487:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787830026.435495:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830026/real 1787830026] req@ffff9e680a2d7c00 x1874668790840192/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830026.435505:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7c00 x1874668790840192/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830026.435511:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830026.435513:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830026.435515:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790840192:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787830026.945827:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787830027.138010:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830027.970031:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787830028.162129:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830028.993823:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787830029.186074:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830030.017822:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787830030.209944:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830031.041976:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787830031.233954:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830031.553856:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830031.553865:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830031.553870:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830031.553880:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6668 00000100:00080000:3.0:1787830031.553891:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830031.553898:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:3.0:1787830031.553931:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d61c0 x1874668790840320/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830031.553970:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830031.553973:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830031.553975:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830031.553978:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6668 00000100:00080000:3.0:1787830031.553982:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830031.553985:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787830031.553994:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830031.554004:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d52c0 x1874668790840448/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830031.554018:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830031.554021:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830031.554023:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830031.554026:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6668 00000100:00080000:3.0:1787830031.554029:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830031.554031:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787830031.554034:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830031.554041:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d7480 x1874668790840576/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830031.554047:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830031.554049:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830031.554050:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830031.554052:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6668 00000100:00080000:3.0:1787830031.554053:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830031.554055:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787830031.554058:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830031.554062:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4b40 x1874668790840704/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830031.554068:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830031.554070:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830031.554071:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830031.554072:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6668 00000100:00080000:3.0:1787830031.554074:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830031.554076:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787830031.554078:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830031.554083:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4780 x1874668790840832/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830031.554088:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830031.554089:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830031.554090:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830031.554092:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6668 00000100:00080000:3.0:1787830031.554093:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830031.554095:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787830031.554097:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830031.554102:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d70c0 x1874668790840960/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830031.554106:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830031.554107:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787830031.554108:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830031.554110:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6668 00000100:00080000:3.0:1787830031.554111:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830031.554113:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787830031.554115:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830031.554119:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d6580 x1874668790841088/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830031.554127:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830031.554129:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830031.554130:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830031.554132:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6668 00000100:00080000:3.0:1787830031.554133:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830031.554135:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787830031.554137:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830031.554141:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d4000 x1874668790841216/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830031.554146:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830031.554147:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830031.554148:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830031.554150:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6668 00000100:00080000:3.0:1787830031.554151:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830031.554153:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:3.0:1787830031.554155:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787830031.554160:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e680a2d43c0 x1874668790841344/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830031.554313:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790840320:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787830031.554404:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790840448:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787830031.554433:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790840576:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787830031.554456:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790840704:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787830031.554476:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790840832:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787830031.554503:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790840960:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787830031.554517:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790841088:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787830031.554529:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790841216:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787830031.554541:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790841344:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787830032.065781:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787830032.258392:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830033.090020:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787830033.281845:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830034.113824:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787830034.306090:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787830034.626096:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.109:988 00000400:00000100:1.0:1787830034.626389:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.109@tcp at host 192.168.203.109 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787830034.626411:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787830034.627883:0:2386:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000800:00000100:1.0:1787830034.627930:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787830034.627959:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787830034.627969:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787830034.627978:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787830034.627987:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787830034.627997:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000800:00000100:1.0:1787830034.628005:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.109@tcp 00000100:00100000:0.0:1787830034.628425:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830031/real 1787830034] req@ffff9e680a2d61c0 x1874668790840320/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830034.628472:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d61c0 x1874668790840320/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830034.628486:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830034.628491:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830034.628496:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d61c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790840320:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787830034.628504:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830031/real 1787830034] req@ffff9e680a2d52c0 x1874668790840448/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830034.628517:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d52c0 x1874668790840448/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830034.628530:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830034.628534:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830034.628537:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d52c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790840448:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787830034.628544:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830031/real 1787830034] req@ffff9e680a2d7480 x1874668790840576/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830034.628555:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d7480 x1874668790840576/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830034.628562:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830034.628566:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830034.628569:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790840576:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787830034.628576:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830031/real 1787830034] req@ffff9e680a2d4b40 x1874668790840704/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830034.628586:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4b40 x1874668790840704/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830034.628593:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830034.628596:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830034.628599:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790840704:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787830034.628605:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830031/real 1787830034] req@ffff9e680a2d4780 x1874668790840832/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830034.629337:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4780 x1874668790840832/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830034.629349:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830034.629354:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830034.629358:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790840832:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787830034.629369:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830031/real 1787830034] req@ffff9e680a2d70c0 x1874668790840960/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830034.629382:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d70c0 x1874668790840960/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830034.629393:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830034.629397:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830034.629400:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790840960:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787830034.629408:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830031/real 1787830034] req@ffff9e680a2d6580 x1874668790841088/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830034.629420:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d6580 x1874668790841088/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830034.629427:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830034.629431:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830034.629434:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790841088:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787830034.629441:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830031/real 1787830034] req@ffff9e680a2d4000 x1874668790841216/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830034.629452:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d4000 x1874668790841216/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830034.629460:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830034.629476:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830034.629480:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790841216:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787830034.629487:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830031/real 1787830034] req@ffff9e680a2d43c0 x1874668790841344/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830034.629500:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e680a2d43c0 x1874668790841344/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830034.629509:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830034.629513:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830034.629516:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e680a2d43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790841344:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787830035.137891:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787830035.330635:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830036.161835:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787830036.354495:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830036.673903:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830036.673915:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830036.673919:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830036.673929:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6673 00000100:00080000:2.0:1787830036.673940:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830036.673948:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787830036.673986:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420780 x1874668790841472/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830036.674015:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830036.674018:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830036.674020:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830036.674024:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6673 00000100:00080000:2.0:1787830036.674028:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830036.674031:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787830036.674042:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830036.674056:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044221c0 x1874668790841600/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830036.674065:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790841472:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787830036.674070:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830036.674074:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830036.674076:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830036.674080:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6673 00000100:00080000:2.0:1787830036.674084:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830036.674088:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787830036.674093:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830036.674104:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422940 x1874668790841728/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830036.674113:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830036.674116:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830036.674118:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830036.674124:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6673 00000800:00000100:0.0:1787830036.674125:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00080000:2.0:1787830036.674130:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830036.674136:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787830036.674141:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830036.674151:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422d00 x1874668790841856/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830036.674160:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830036.674163:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830036.674166:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787830036.674171:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830036/real 1787830036] req@ffff9e6804420780 x1874668790841472/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830036.674171:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6673 00000100:00080000:2.0:1787830036.674175:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830036.674182:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787830036.674189:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787830036.674199:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420780 x1874668790841472/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830036.674203:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790841984/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830036.674218:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830036.674220:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830036.674225:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830036.674226:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00080000:2.0:1787830036.674228:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787830036.674233:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790841472:192.168.203.109@tcp:250:kworker.0 00000100:00080000:2.0:1787830036.674234:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6673 00000100:00080000:2.0:1787830036.674239:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830036.674245:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787830036.674249:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830036.674260:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790842112/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830036.674262:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790841600:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787830036.674277:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830036.674280:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830036.674282:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830036.674287:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6673 00000100:00080000:2.0:1787830036.674299:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830036.674304:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787830036.674309:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787830036.674310:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:2.0:1787830036.674324:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790842240/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830036.674330:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830036/real 1787830036] req@ffff9e68044221c0 x1874668790841600/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830036.674338:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830036.674345:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830036.674347:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787830036.674352:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044221c0 x1874668790841600/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830036.674352:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6673 00000100:00080000:2.0:1787830036.674358:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830036.674365:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00080000:0.0:1787830036.674371:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787830036.674371:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787830036.674378:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830036.674383:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044221c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790841600:192.168.203.109@tcp:38:kworker.0 00000100:00100000:2.0:1787830036.674386:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790842368/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830036.674393:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790841728:192.168.203.109@tcp:38:kworker.0 00000100:00080000:2.0:1787830036.674399:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830036.674403:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830036.674405:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830036.674409:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6673 00000100:00080000:2.0:1787830036.674413:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830036.674416:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787830036.674421:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830036.674430:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790842496/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787830036.674500:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787830036.674511:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830036/real 1787830036] req@ffff9e6804422940 x1874668790841728/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830036.674524:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422940 x1874668790841728/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830036.674532:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830036.674535:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830036.674538:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790841728:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787830036.674544:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790841856:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787830036.674563:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787830036.674573:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830036/real 1787830036] req@ffff9e6804422d00 x1874668790841856/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830036.674585:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422d00 x1874668790841856/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830036.674592:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830036.674595:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830036.674598:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790841856:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787830036.674603:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790841984:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787830036.674619:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787830036.674629:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830036/real 1787830036] req@ffff9e6804423840 x1874668790841984/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830036.674640:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790841984/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830036.674652:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830036.674655:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830036.674658:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790841984:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787830036.674674:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790842112:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787830036.674694:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787830036.674736:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830036/real 1787830036] req@ffff9e6804423480 x1874668790842112/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830036.674747:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790842112/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830036.674756:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830036.674758:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830036.674761:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790842112:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787830036.674767:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790842240:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787830036.674784:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787830036.674794:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830036/real 1787830036] req@ffff9e6804420f00 x1874668790842240/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830036.674803:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790842240/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830036.674810:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830036.674812:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830036.674816:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790842240:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787830036.674821:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790842368:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787830036.674835:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787830036.674845:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830036/real 1787830036] req@ffff9e6804422580 x1874668790842368/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830036.674855:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790842368/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830036.674862:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830036.674865:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830036.674868:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790842368:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787830036.674873:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790842496:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787830036.674890:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787830036.674900:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830036/real 1787830036] req@ffff9e6804421e00 x1874668790842496/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830036.674910:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790842496/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830036.674917:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830036.674920:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830036.674923:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790842496:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787830037.185821:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787830037.377997:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830038.210090:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787830038.402680:0:6168:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830039.233947:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787830039.427163:0:14926:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830040.257874:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:3.0:1787830040.449954:0:6167:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830041.282291:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds 00010000:00010000:1.0:1787830041.474780:0:12246:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830041.793857:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) 6bde37df-3766-41d5-a558-be13e58235fe->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830041.793864:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830041.793867:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830041.793876:0:18639:0:(import.c:510:import_select_connection()) MGC192.168.203.109@tcp: connect to NID 192.168.203.109@tcp last attempt 6678 00000100:00080000:2.0:1787830041.793885:0:18639:0:(import.c:589:import_select_connection()) MGC192.168.203.109@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830041.793891:0:18639:0:(import.c:624:import_select_connection()) MGC192.168.203.109@tcp: import ffff9e68181c6800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000100:00100000:2.0:1787830041.793921:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044212c0 x1874668790842624/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830041.793951:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830041.793955:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830041.793957:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830041.793961:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6678 00000100:00080000:2.0:1787830041.793964:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830041.793967:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e68205f2800: import ffff9e6810101800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787830041.793976:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830041.793988:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e68044230c0 x1874668790842752/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830041.793997:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830041.793999:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830041.794001:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830041.794004:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6678 00000100:00080000:2.0:1787830041.794008:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830041.794011:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e68205f2800: import ffff9e6810107800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787830041.794016:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830041.794025:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420b40 x1874668790842880/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830041.794039:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830041.794042:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830041.794044:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830041.794047:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6678 00000100:00080000:2.0:1787830041.794050:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830041.794053:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e68205f2800: import ffff9e68181de800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787830041.794058:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830041.794068:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421680 x1874668790843008/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830041.794076:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) bb930b33-002c-4632-a90b-528a6bdcd03e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830041.794079:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830041.794081:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830041.794084:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: connect to NID 192.168.203.109@tcp last attempt 6678 00000100:00080000:2.0:1787830041.794087:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830041.794090:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e68205f2800: import ffff9e68181df800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787830041.794094:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830041.794103:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804421e00 x1874668790843136/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830041.794111:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830041.794114:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830041.794116:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830041.794119:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6678 00000100:00080000:2.0:1787830041.794123:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830041.794127:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9e6806580800: import ffff9e6809276800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787830041.794131:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830041.794156:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804422580 x1874668790843264/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830041.794165:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830041.794168:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787830041.794169:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830041.794173:0:18639:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6678 00000100:00080000:2.0:1787830041.794176:0:18639:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830041.794180:0:18639:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9e6806580800: import ffff9e6809276000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787830041.794184:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830041.794193:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804420f00 x1874668790843392/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830041.794202:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830041.794204:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830041.794206:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830041.794209:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6678 00000100:00080000:2.0:1787830041.794212:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830041.794216:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9e6806580800: import ffff9e68205f1800 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787830041.794220:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830041.794228:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423480 x1874668790843520/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830041.794238:0:18639:0:(pinger.c:229:ptlrpc_pinger_process_import()) fa787581-6963-4f39-b38f-836255c71d7c->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830041.794240:0:18639:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830041.794241:0:18639:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830041.794244:0:18639:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: connect to NID 192.168.203.109@tcp last attempt 6678 00000100:00080000:2.0:1787830041.794245:0:18639:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830041.794248:0:18639:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9e6806580800: import ffff9e68052ed000 using connection 192.168.203.109@tcp/192.168.203.109@tcp 00000008:00100000:2.0:1787830041.794250:0:18639:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787830041.794255:0:18639:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9e6804423840 x1874668790843648/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830041.794355:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790842624:192.168.203.109@tcp:250:kworker.0 00000800:00000100:0.0:1787830041.794406:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787830041.794463:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830041/real 1787830041] req@ffff9e68044212c0 x1874668790842624/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830041.794483:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044212c0 x1874668790842624/t0(0) o250->MGC192.168.203.109@tcp@192.168.203.109@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830041.794493:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181c6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830041.794497:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830041.794501:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044212c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:6bde37df-3766-41d5-a558-be13e58235fe:2398:1874668790842624:192.168.203.109@tcp:250:kworker.0 00000100:00100000:0.0:1787830041.794508:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790842752:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787830041.794527:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787830041.794537:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830041/real 1787830041] req@ffff9e68044230c0 x1874668790842752/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830041.794544:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e68044230c0 x1874668790842752/t0(0) o38->lustre-MDT0000-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830041.794550:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810101800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830041.794551:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830041.794554:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e68044230c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790842752:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787830041.794557:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790842880:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787830041.794568:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787830041.794573:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830041/real 1787830041] req@ffff9e6804420b40 x1874668790842880/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830041.794580:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420b40 x1874668790842880/t0(0) o38->lustre-MDT0001-mdc-ffff9e68205f2800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830041.794588:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6810107800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830041.794590:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830041.794591:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790842880:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787830041.794594:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790843008:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787830041.794603:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787830041.794608:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830041/real 1787830041] req@ffff9e6804421680 x1874668790843008/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830041.794614:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421680 x1874668790843008/t0(0) o8->lustre-OST0000-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830041.794618:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181de800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830041.794620:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830041.794622:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790843008:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787830041.794625:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790843136:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787830041.794633:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787830041.794638:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830041/real 1787830041] req@ffff9e6804421e00 x1874668790843136/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830041.794643:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804421e00 x1874668790843136/t0(0) o8->lustre-OST0001-osc-ffff9e68205f2800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830041.794647:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68181df800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830041.794649:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830041.794651:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804421e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:bb930b33-002c-4632-a90b-528a6bdcd03e:2398:1874668790843136:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787830041.794654:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790843264:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787830041.794663:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787830041.794668:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830041/real 1787830041] req@ffff9e6804422580 x1874668790843264/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830041.794674:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804422580 x1874668790843264/t0(0) o38->lustre-MDT0000-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830041.794679:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830041.794681:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830041.794682:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804422580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790843264:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787830041.794685:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790843392:192.168.203.109@tcp:38:kworker.0 00000800:00000100:0.0:1787830041.794693:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787830041.794723:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830041/real 1787830041] req@ffff9e6804420f00 x1874668790843392/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830041.794729:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804420f00 x1874668790843392/t0(0) o38->lustre-MDT0001-mdc-ffff9e6806580800@192.168.203.109@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830041.794734:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e6809276000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830041.794736:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830041.794737:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804420f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790843392:192.168.203.109@tcp:38:kworker.0 00000100:00100000:0.0:1787830041.794740:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790843520:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787830041.794751:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787830041.794756:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830041/real 1787830041] req@ffff9e6804423480 x1874668790843520/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830041.794762:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423480 x1874668790843520/t0(0) o8->lustre-OST0000-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830041.794767:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68205f1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830041.794768:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830041.794770:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790843520:192.168.203.109@tcp:8:kworker.0 00000100:00100000:0.0:1787830041.794773:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790843648:192.168.203.109@tcp:8:kworker.0 00000800:00000100:0.0:1787830041.794781:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.109@tcp 00000100:00100000:0.0:1787830041.794786:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830041/real 1787830041] req@ffff9e6804423840 x1874668790843648/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830041.794792:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9e6804423840 x1874668790843648/t0(0) o8->lustre-OST0001-osc-ffff9e6806580800@192.168.203.109@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830041.794796:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9e68052ed000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830041.794798:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.109@tcp failed (-110) 00000100:00100000:0.0:1787830041.794799:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9e6804423840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:fa787581-6963-4f39-b38f-836255c71d7c:2398:1874668790843648:192.168.203.109@tcp:8:kworker.0 00000100:00100000:3.0:1787830042.305843:0:88801:0:(client.c:2715:ptlrpc_set_wait()) set ffff9e68037d3f00 going to sleep for 0 seconds Debug log: 76599 lines, 76599 kept, 0 dropped, 0 bad.